Document reclaiming disk, and what it deliberately will not reclaim
CI / test (push) Successful in 1m49s
CI / test (push) Successful in 1m49s
A command that deletes data needs its limits written down more than its syntax. The section says plainly that llmgw only takes back files it fetched itself, and then says the uncomfortable half out loud: that this means it will not reclaim the storage most likely to be filling your disk, because weights you placed yourself are yours to remove. Also records the dash in the DISK column as "not asked" rather than "no space", which is the reading that matters when a node will not schedule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e6bc7859dc
commit
1b7e11b91e
2 files changed
+79
No files matched your search
@@ -595,6 +595,38 @@ fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
## Reclaiming disk
|
||||
|
||||
`llmgw ctl nodes` has a DISK column: free space against the size it is free
|
||||
out of, for the filesystems holding model weights. A dash means the node was
|
||||
not asked — an agent with no `model_dirs`, or one older than this — and not
|
||||
that it has no space. Shared filesystems are excluded from a node's figure,
|
||||
because space on a filer belongs to everyone mounting it.
|
||||
|
||||
Weights llmgw downloaded for models that are no longer in the catalog can be
|
||||
listed and removed:
|
||||
|
||||
```
|
||||
llmgw ctl weights # what could be freed, per node
|
||||
llmgw ctl weights --collect # delete it
|
||||
```
|
||||
|
||||
**Only files llmgw fetched itself are ever candidates.** They are recognisable
|
||||
because llmgw names them after a hash of their source, so the set still wanted
|
||||
is computed from the catalog exactly rather than guessed. Anything you put on
|
||||
a node — by scp, by a shared mount, by hand — is left alone however
|
||||
unreferenced it looks, and no llmgw command will remove it. The nodes check
|
||||
every path again before deleting it, so a control plane that is wrong cannot
|
||||
talk one into losing a model that took a week to copy.
|
||||
|
||||
Which also means llmgw will not reclaim the storage most likely to be filling
|
||||
your disk. Weights you placed yourself are yours to remove.
|
||||
|
||||
A profile may declare `min_disk_gb`, and a node without that much free is then
|
||||
not a candidate for it — checked only on nodes that do not already hold the
|
||||
weights, since those need no room for them. Without the field, disk is not
|
||||
considered, which is how every profile written before it keeps behaving.
|
||||
|
||||
## When something does not work
|
||||
|
||||
**The node is `ready` but every replica fails.** Almost always a missing engine
|
||||
|
||||
Reference in new issue
Block a user