The default arrangement is the simple one: every node keeps its own copy. So a source naming a remote file is now fetched before the engine is launched, rather than left to fail as a missing path minutes into a load. Eight shards of a tensor-parallel replica start at the same instant and want the same file; they produce one download. An interrupted fetch resumes instead of starting over, which matters at forty gigabytes, and the download is written beside its target and renamed only when complete — a partial file under the model's name would be found by the next scan, reported as present, and handed to an engine that fails on it. The exotic arrangements need no code, which is the point. Weights on a Lustre mount or an image exported over the fabric are simply present, and present means nothing is fetched. The escape hatch from a distribution mechanism is a distribution mechanism that checks first. Wiring it up caught a regression in my own design. A bare model id — "Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang resolve themselves, and treating every non-remote source as a local path made every existing vLLM profile fail. Only an absolute or explicitly relative path is now checked as a file; anything else is passed through untouched. Three agent tests failed on the same change for a better reason: they used "/models/x", and the new check is exactly the one that catches that. Whole repositories of shards are deliberately not fetched here. vLLM and SGLang do that well already, and a second snapshot downloader would be work spent arriving back where we started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
c13c567cc5
commit
943c23a412
5 files changed
+723
-1
No files matched your search
@@ -282,6 +282,35 @@ sudo install -d -o root -g root /var/lib/llmgw-models
|
||||
For a first model, something small: a 3B at `q4_k_m` is about 2 GB, runs on
|
||||
CPU, and answers fast enough to prove the whole path works.
|
||||
|
||||
### Letting llmgw fetch them
|
||||
|
||||
The simple arrangement, and the default: every node keeps its own copy. Point a
|
||||
model's `source` at a file to download and the agent fetches it once, before the
|
||||
engine is launched:
|
||||
|
||||
```yaml
|
||||
source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
|
||||
# or an ordinary URL
|
||||
source: https://models.internal/qwen2.5-coder-3b-q4_k_m.gguf
|
||||
```
|
||||
|
||||
Eight shards of one replica starting together produce one download, not eight.
|
||||
An interrupted fetch resumes rather than starting over, and a partial file is
|
||||
never visible under the model's name — so a half-downloaded model cannot be
|
||||
picked up by the next scan and handed to an engine.
|
||||
|
||||
Three kinds of source, and only one of them is fetched:
|
||||
|
||||
| `source` | What happens |
|
||||
| --- | --- |
|
||||
| `/var/lib/llmgw-models/x.gguf` | Checked, never copied. However it got there. |
|
||||
| `hf:org/repo/file.gguf`, `https://…` | Fetched once into the cache, then reused |
|
||||
| `Qwen/Qwen2.5-7B-Instruct` | Left alone — vLLM and SGLang download repositories themselves |
|
||||
|
||||
Which means the exotic arrangements need no configuration at all. Weights on a
|
||||
Lustre mount, or an image exported over the fabric with SRP or NVMe-oF, are
|
||||
simply *present*, and present means nothing is fetched.
|
||||
|
||||
### Telling llmgw where they are
|
||||
|
||||
Optional, and unset by default. Naming the directories in the agent's
|
||||
|
||||
Reference in new issue
Block a user