diff --git a/deploy/mirror/README.md b/deploy/mirror/README.md index 4570a8df..14ac886c 100644 --- a/deploy/mirror/README.md +++ b/deploy/mirror/README.md @@ -30,7 +30,61 @@ repository that changes upstream changes what you are running, and the first sign is a prompt that worked last week answering differently this week with nobody able to say why. Pin it here and that cannot happen. -## nginx +## Running it + +`llmgw` is the mirror. Same binary as the control plane and the agents, one +subcommand: + +``` +sudo mkdir -p /srv/models && sudo chown llmgw:llmgw /srv/models +llmgw mirror serve --dir /srv/models --base http://models.internal:8081 +``` + +There is a unit in `deploy/systemd/llmgw-mirror.service`. Set `--base` to the +address nodes reach this machine on — not `localhost`, and not a name only this +host knows itself by. It is the string that ends up in catalog entries, and a +wrong one makes `mirror unused` report files as unreferenced when they are in +use. The next step after that list is deletion. + +### Two listeners + +`--listen :8081` serves the weights. Read-only, unauthenticated, Range requests +supported so an interrupted fetch on a node resumes rather than restarts. This +is the path eight nodes pull forty gigabytes from. + +`--api-listen :8082` is the management API. Separate on purpose: putting a +client certificate in front of the byte path would mean re-certifying every +agent's fetcher in order to change how models are administered. + +The API is unauthenticated unless you give it the fleet's CA, and it says so at +startup rather than leaving you to find out. Anyone who can reach that port can +start downloads and delete models, so either keep it on a trusted network or +hand it certificates: + +``` +llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \ + --tls-ca /etc/llmgw/ca.pem \ + --tls-cert /etc/llmgw/mirror.pem \ + --tls-key /etc/llmgw/mirror-key.pem +``` + +That is the same CA the agents already use, so the certificates come from the +authority you have: + +``` +llmgw ctl cert issue-server mirror --host models.internal --host 10.0.0.146 +llmgw ctl cert issue-client control-plane +``` + +`--host` has to cover every address the API is called on, because that is what +a client verifies against. Both sides are checked — the mirror will not accept a +caller without a certificate from this CA, the same rule the agent stream +follows. + +### nginx instead + +If you already run nginx, it serves a directory perfectly well and you do not +need `mirror serve` for the bytes: ```nginx server { @@ -57,15 +111,81 @@ need it. ## Filling it +From the mirror itself: + ``` -# On the mirror, once per model and version: -mkdir -p /srv/models/qwen2.5-14b-instruct/2026-03 -curl -fL -o /srv/models/qwen2.5-14b-instruct/2026-03/q4_k_m.gguf \ - "https://huggingface.co//resolve//" +llmgw mirror add \ + --model qwen2.5-14b-instruct --version 2026-03 \ + "https://huggingface.co//resolve//q4_k_m.gguf" ``` Use a revision rather than `main` in the source URL. `main` is what moves. +It downloads to a temporary name and renames when complete, so an interrupted +download never leaves something a catalog could be pointed at, and it prints the +sha256 when it lands. Compare that against what upstream publishes — Hugging +Face gives it as `lfs.oid` from +`https://huggingface.co/api/models//tree/`. A mirror is the one +place in this system where a corrupt file is copied to every node instead of +being caught once. + +It refuses to overwrite a version that is already there. Add a new one beside +it: that refusal is what makes the old weights still available when the new ones +turn out to be worse. + +## Driving it from elsewhere + +The same fetch over the API, so the control plane — or anything else — can ask +for a model without anyone logging in: + +``` +curl -X POST http://models.internal:8082/v1/models -d '{ + "source": "https://huggingface.co//resolve//q4_k_m.gguf", + "model": "qwen2.5-14b-instruct", + "version": "2026-03" +}' +``` + +The mirror does the downloading. A control plane that pulled the bytes itself +and pushed them here would carry every model twice over the network, through the +one machine least able to spare the bandwidth. + +It answers `202` with the URL a catalog entry should use, and the bytes are not +here yet. Poll for them: + +| | | +|---|---| +| `GET /v1/models` | everything held or arriving, and the base to build URLs from | +| `POST /v1/models` | fetch this source into model/version | +| `GET /v1/models/{model}/{version}` | `fetching` with a byte count, `present`, or `failed` with the reason | +| `DELETE /v1/models/{model}/{version}` | remove a version; refuses while it is downloading | +| `GET /v1/health` | up *and* able to read its own directory | + +A failed fetch stays `failed` with what went wrong, rather than disappearing: +otherwise the thing waiting on that model waits forever with nothing to say why. +Asking twice for the same download joins the one already running instead of +starting a rival writer on the same file. + +`DELETE` takes the whole version, never one file of it. A model split across +several files is useless with one missing, and half a version on disk looks +present and will not load. + +## Retiring a model + +``` +llmgw mirror unused # what no catalog entry refers to +llmgw mirror remove --model --version --file +``` + +`unused` compares against every `source` in the running catalog, so the answer +is what can go rather than what looks old. It refuses to answer when it cannot +reach the control plane: an empty catalog and an unreachable one produce the +same list — everything — and the obvious next step after that list is deletion. + +Retire the catalog entry first, let the fleet settle, then remove. A node that +has to refetch a file you deleted cannot, and the failure arrives later, on the +node, looking like something else. + ## Pointing llmgw at it ```yaml @@ -86,11 +206,11 @@ model's name, so an interrupted fetch cannot be handed to an engine. ## What this is not -It is not a cache: nothing populates it automatically, and a model you did not -put there is not there. That is on purpose. A mirror that fetches on demand is -a mirror that reaches the internet at the least convenient moment, and in a -network where that is blocked it fails in a way that looks like llmgw being -broken. +It is not a cache. Nothing populates it on its own: a model is there because +somebody asked for it, by hand or over the API, and one nobody asked for is not +there. That is on purpose. A mirror that fetched on demand would reach the +internet at the least convenient moment, and in a network where that is blocked +it would fail in a way that looks like llmgw being broken. It is also not a replacement for a shared filesystem. If you already run one that every node mounts, put the models on it and use a path as the `source` — diff --git a/deploy/systemd/llmgw-mirror.service b/deploy/systemd/llmgw-mirror.service new file mode 100644 index 00000000..62915e92 --- /dev/null +++ b/deploy/systemd/llmgw-mirror.service @@ -0,0 +1,29 @@ +[Unit] +Description=llmgw model mirror +After=network-online.target +Wants=network-online.target + +[Service] +# --base is what a catalog entry will say, so it has to be the address nodes +# reach this machine on — not localhost, and not the name only this host knows +# itself by. Getting it wrong makes `mirror unused` report files as unreferenced +# when they are in use, and the next step after that list is deletion. +ExecStart=/usr/local/bin/llmgw mirror serve \ + --dir /srv/models \ + --listen :8081 \ + --api-listen :8082 \ + --base http://models.internal:8081 +User=llmgw +Group=llmgw +Restart=on-failure +RestartSec=2s +NoNewPrivileges=true +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +# The one writable path. The mirror serves reads to the whole fleet and writes +# only what its own API was asked to fetch, so nothing else needs to be. +ReadWritePaths=/srv/models + +[Install] +WantedBy=multi-user.target