Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s

Storage will be different at every installation — Lustre, NFS over RDMA, an
image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes
it rather than managing it. An agent told which directories to look in
reports what it can see; how the files got there is nobody's business here,
and a node told nothing reports nothing and behaves exactly as every node
did before this existed.

That fact buys two things. Placement prefers a node that already holds the
model over one that must first move tens of gigabytes, as a preference and
never a filter: fetching them is a valid answer, and refusing to place would
turn a slow start into no start at all. And ctl status shows what each
fabric holds.

Per fabric because fabrics are not connected to one another. Weights in
ib-a are not weights in ib-b — the only path between them is the management
network — so a model seeded into one fabric leaves the rest of the cluster
unable to serve it without a slow copy, and nothing said so before now. One
shared filer counts the same as twenty local copies, because "can this
fabric serve it" is the question being asked.

Three distinctions the tests hold to, because each is a way to get this
quietly wrong. A node that was never asked to look must rank with one that
looked and has nothing, or enabling the feature on a single node would
drain every other. An empty directory is not weights, because a
half-finished download looks exactly like one and would send placement to a
node whose engine then fails to load. And local storage is never mistaken
for shared: a filer read as local costs an unnecessary copy, while local
read as shared would have the cluster believe a model is everywhere when it
is on one node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 08:21:16 +02:00
1 parent bd134a7ab9
commit c13c567cc5
18 files changed
+782 -63

No files matched your search

+28
View File
@@ -282,6 +282,34 @@ sudo install -d -o root -g root /var/lib/llmgw-models
For a first model, something small: a 3B at `q4_k_m` is about 2 GB, runs on
CPU, and answers fast enough to prove the whole path works.
### Telling llmgw where they are
Optional, and unset by default. Naming the directories in the agent's
`model_dirs` lets a node report which models it can see, and placement then
prefers a node that already holds one over a node that must first move tens of
gigabytes:
```yaml
model_dirs:
- /var/lib/llmgw-models
- /mnt/lustre/models
```
llmgw does not manage that storage and does not care how the files arrived. A
Lustre or NFS mount, an image exported over the fabric with SRP or NVMe-oF,
local NVMe, or an `scp` all look identical to it. A node with nothing configured
reports nothing and is treated exactly as every node was before this existed.
**It matters most when your fabrics are not connected to each other.** Weights
in `ib-a` are not weights in `ib-b`: the only path between them is the
management network, which is the slow one. `llmgw ctl status` shows what each
fabric holds, so a model seeded into one fabric does not quietly leave the rest
of the cluster unable to serve it.
How much disk a node needs follows from the models rather than a rule of thumb.
`llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
bf16 is 141 GB and the same model at q4 is 43 GB.
## 5. Adding the node
Two halves, because they need different access: building the machine needs the