Fetch a model's weights when the simple arrangement needs it
CI / test (push) Successful in 1m26s

The default arrangement is the simple one: every node keeps its own copy.
So a source naming a remote file is now fetched before the engine is
launched, rather than left to fail as a missing path minutes into a load.

Eight shards of a tensor-parallel replica start at the same instant and
want the same file; they produce one download. An interrupted fetch resumes
instead of starting over, which matters at forty gigabytes, and the
download is written beside its target and renamed only when complete — a
partial file under the model's name would be found by the next scan,
reported as present, and handed to an engine that fails on it.

The exotic arrangements need no code, which is the point. Weights on a
Lustre mount or an image exported over the fabric are simply present, and
present means nothing is fetched. The escape hatch from a distribution
mechanism is a distribution mechanism that checks first.

Wiring it up caught a regression in my own design. A bare model id —
"Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang
resolve themselves, and treating every non-remote source as a local path
made every existing vLLM profile fail. Only an absolute or explicitly
relative path is now checked as a file; anything else is passed through
untouched. Three agent tests failed on the same change for a better reason:
they used "/models/x", and the new check is exactly the one that catches
that.

Whole repositories of shards are deliberately not fetched here. vLLM and
SGLang do that well already, and a second snapshot downloader would be work
spent arriving back where we started.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 08:46:03 +02:00
1 parent c13c567cc5
commit 943c23a412
5 files changed
+723 -1

No files matched your search

+29
View File
@@ -282,6 +282,35 @@ sudo install -d -o root -g root /var/lib/llmgw-models
For a first model, something small: a 3B at `q4_k_m` is about 2 GB, runs on
CPU, and answers fast enough to prove the whole path works.
### Letting llmgw fetch them
The simple arrangement, and the default: every node keeps its own copy. Point a
model's `source` at a file to download and the agent fetches it once, before the
engine is launched:
```yaml
source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
# or an ordinary URL
source: https://models.internal/qwen2.5-coder-3b-q4_k_m.gguf
```
Eight shards of one replica starting together produce one download, not eight.
An interrupted fetch resumes rather than starting over, and a partial file is
never visible under the model's name — so a half-downloaded model cannot be
picked up by the next scan and handed to an engine.
Three kinds of source, and only one of them is fetched:
| `source` | What happens |
| --- | --- |
| `/var/lib/llmgw-models/x.gguf` | Checked, never copied. However it got there. |
| `hf:org/repo/file.gguf`, `https://…` | Fetched once into the cache, then reused |
| `Qwen/Qwen2.5-7B-Instruct` | Left alone — vLLM and SGLang download repositories themselves |
Which means the exotic arrangements need no configuration at all. Weights on a
Lustre mount, or an image exported over the fabric with SRP or NVMe-oF, are
simply *present*, and present means nothing is fetched.
### Telling llmgw where they are
Optional, and unset by default. Naming the directories in the agent's