Document reclaiming disk, and what it deliberately will not reclaim
CI / test (push) Successful in 1m49s

A command that deletes data needs its limits written down more than its
syntax. The section says plainly that llmgw only takes back files it fetched
itself, and then says the uncomfortable half out loud: that this means it will
not reclaim the storage most likely to be filling your disk, because weights
you placed yourself are yours to remove.

Also records the dash in the DISK column as "not asked" rather than "no
space", which is the reading that matters when a node will not schedule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-18 09:59:16 +02:00
1 parent e6bc7859dc
commit 1b7e11b91e
2 files changed
+79

No files matched your search

+32
View File
@@ -595,6 +595,38 @@ fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.
## Reclaiming disk
`llmgw ctl nodes` has a DISK column: free space against the size it is free
out of, for the filesystems holding model weights. A dash means the node was
not asked — an agent with no `model_dirs`, or one older than this — and not
that it has no space. Shared filesystems are excluded from a node's figure,
because space on a filer belongs to everyone mounting it.
Weights llmgw downloaded for models that are no longer in the catalog can be
listed and removed:
```
llmgw ctl weights # what could be freed, per node
llmgw ctl weights --collect # delete it
```
**Only files llmgw fetched itself are ever candidates.** They are recognisable
because llmgw names them after a hash of their source, so the set still wanted
is computed from the catalog exactly rather than guessed. Anything you put on
a node — by scp, by a shared mount, by hand — is left alone however
unreferenced it looks, and no llmgw command will remove it. The nodes check
every path again before deleting it, so a control plane that is wrong cannot
talk one into losing a model that took a week to copy.
Which also means llmgw will not reclaim the storage most likely to be filling
your disk. Weights you placed yourself are yours to remove.
A profile may declare `min_disk_gb`, and a node without that much free is then
not a candidate for it — checked only on nodes that do not already hold the
weights, since those need no room for them. Without the field, disk is not
considered, which is how every profile written before it keeps behaving.
## When something does not work
**The node is `ready` but every replica fails.** Almost always a missing engine