Document reclaiming disk, and what it deliberately will not reclaim
CI / test (push) Successful in 1m49s

A command that deletes data needs its limits written down more than its
syntax. The section says plainly that llmgw only takes back files it fetched
itself, and then says the uncomfortable half out loud: that this means it will
not reclaim the storage most likely to be filling your disk, because weights
you placed yourself are yours to remove.

Also records the dash in the DISK column as "not asked" rather than "no
space", which is the reading that matters when a node will not schedule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-18 09:59:16 +02:00
1 parent e6bc7859dc
commit 1b7e11b91e
2 files changed
+79

No files matched your search

+47
View File
@@ -16,6 +16,53 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
### Disk, and taking back only what llmgw put there
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space in the same struct was discarded, so the only way to find
out why a node was filling up was to ssh to it.
Nodes now report it, per filesystem rather than per configured directory: free
space belongs to the disk, and two model directories on one disk are one pool
of bytes. Shared filesystems are kept out of a node's own figure, because space
on a filer belongs to everyone mounting it and twenty nodes would otherwise
each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
Prometheus, and a node that reported nothing reads as unknown everywhere — an
older agent must not look like a full disk.
A profile may declare `min_disk_gb`, and a node without the room is no longer a
candidate, with a reason `explain-placement` prints. Skipped on a node that
already holds the weights: it needs no room for them, and without that the node
best placed to run a model is the one most likely to be rejected.
`ctl weights` lists model files llmgw downloaded for models no longer in the
catalog, and `--collect` deletes them. The design is mostly about what it will
not do. Only files llmgw fetched are candidates, and that is a calculation
rather than a heuristic — the cache name is a hash of the source, so the wanted
set is computable exactly. Anything an operator put on a node is left alone
however unreferenced it looks. Each node checks every path again before
unlinking it, refusing anything outside its cache directory or without the
cache name, so a control plane that is wrong cannot talk a node into losing a
model that took a week to copy.
### Fixes
- **A capability was lost at an indirection.** The controller is handed a
forwarding dispatcher, and reaches past its interface by type assertion to
find what lies beyond scheduling. Deleting weights was implemented on the
agent service and not on the forwarder, so it was simply absent at runtime —
with the whole suite green, because the dispatchers the tests use do not
implement it either and its absence there is the expected case.
- **`ctl tool-check` blamed the engine for what the model did.** It had one
verdict for a failed attempt and told operators not to try another model.
Measured: the engine renders the tool prompt correctly, the coder models do
not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
the same engine returns a populated `tool_calls` array. A failed attempt is
now two verdicts, told apart by whether the reply used the wrapper.
## v0.6.0 — 2026-09-18
Phase 4: a replica can be bigger than one machine, and the fabric it spans
+32
View File
@@ -595,6 +595,38 @@ fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.
## Reclaiming disk
`llmgw ctl nodes` has a DISK column: free space against the size it is free
out of, for the filesystems holding model weights. A dash means the node was
not asked — an agent with no `model_dirs`, or one older than this — and not
that it has no space. Shared filesystems are excluded from a node's figure,
because space on a filer belongs to everyone mounting it.
Weights llmgw downloaded for models that are no longer in the catalog can be
listed and removed:
```
llmgw ctl weights # what could be freed, per node
llmgw ctl weights --collect # delete it
```
**Only files llmgw fetched itself are ever candidates.** They are recognisable
because llmgw names them after a hash of their source, so the set still wanted
is computed from the catalog exactly rather than guessed. Anything you put on
a node — by scp, by a shared mount, by hand — is left alone however
unreferenced it looks, and no llmgw command will remove it. The nodes check
every path again before deleting it, so a control plane that is wrong cannot
talk one into losing a model that took a week to copy.
Which also means llmgw will not reclaim the storage most likely to be filling
your disk. Weights you placed yourself are yours to remove.
A profile may declare `min_disk_gb`, and a node without that much free is then
not a candidate for it — checked only on nodes that do not already hold the
weights, since those need no room for them. Without the field, disk is not
considered, which is how every profile written before it keeps behaving.
## When something does not work
**The node is `ready` but every replica fails.** Almost always a missing engine