From 1b7e11b91e6dff4a6010e48439de6c66db443ff3 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Fri, 18 Sep 2026 09:59:16 +0200 Subject: [PATCH] Document reclaiming disk, and what it deliberately will not reclaim A command that deletes data needs its limits written down more than its syntax. The section says plainly that llmgw only takes back files it fetched itself, and then says the uncomfortable half out loud: that this means it will not reclaim the storage most likely to be filling your disk, because weights you placed yourself are yours to remove. Also records the dash in the DISK column as "not asked" rather than "no space", which is the reading that matters when a node will not schedule. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ docs/install.md | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 71463a9a..c564a46b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,53 @@ is only released once its **exit criterion** is met, not when its code lands. Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period. +## Unreleased + +### Disk, and taking back only what llmgw put there + +Disk was the one resource nothing reported. `statfs` was already being called +on every model directory, but only to ask whether the filesystem was a network +one — the free space in the same struct was discarded, so the only way to find +out why a node was filling up was to ssh to it. + +Nodes now report it, per filesystem rather than per configured directory: free +space belongs to the disk, and two model directories on one disk are one pool +of bytes. Shared filesystems are kept out of a node's own figure, because space +on a filer belongs to everyone mounting it and twenty nodes would otherwise +each report the same free terabyte. It shows in `ctl nodes`, the dashboard and +Prometheus, and a node that reported nothing reads as unknown everywhere — an +older agent must not look like a full disk. + +A profile may declare `min_disk_gb`, and a node without the room is no longer a +candidate, with a reason `explain-placement` prints. Skipped on a node that +already holds the weights: it needs no room for them, and without that the node +best placed to run a model is the one most likely to be rejected. + +`ctl weights` lists model files llmgw downloaded for models no longer in the +catalog, and `--collect` deletes them. The design is mostly about what it will +not do. Only files llmgw fetched are candidates, and that is a calculation +rather than a heuristic — the cache name is a hash of the source, so the wanted +set is computable exactly. Anything an operator put on a node is left alone +however unreferenced it looks. Each node checks every path again before +unlinking it, refusing anything outside its cache directory or without the +cache name, so a control plane that is wrong cannot talk a node into losing a +model that took a week to copy. + +### Fixes + +- **A capability was lost at an indirection.** The controller is handed a + forwarding dispatcher, and reaches past its interface by type assertion to + find what lies beyond scheduling. Deleting weights was implemented on the + agent service and not on the forwarder, so it was simply absent at runtime — + with the whole suite green, because the dispatchers the tests use do not + implement it either and its absence there is the expected case. +- **`ctl tool-check` blamed the engine for what the model did.** It had one + verdict for a failed attempt and told operators not to try another model. + Measured: the engine renders the tool prompt correctly, the coder models do + not use the `` wrapper it asks for, and `qwen2.5-3b-instruct` on + the same engine returns a populated `tool_calls` array. A failed attempt is + now two verdicts, told apart by whether the reply used the wrapper. + ## v0.6.0 — 2026-09-18 Phase 4: a replica can be bigger than one machine, and the fabric it spans diff --git a/docs/install.md b/docs/install.md index 4819b788..3b1b03d5 100644 --- a/docs/install.md +++ b/docs/install.md @@ -595,6 +595,38 @@ fleet means the measurement is missing, not that the fleet is idle. And `llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown, which is what a binary built without the nvml tag reports. +## Reclaiming disk + +`llmgw ctl nodes` has a DISK column: free space against the size it is free +out of, for the filesystems holding model weights. A dash means the node was +not asked — an agent with no `model_dirs`, or one older than this — and not +that it has no space. Shared filesystems are excluded from a node's figure, +because space on a filer belongs to everyone mounting it. + +Weights llmgw downloaded for models that are no longer in the catalog can be +listed and removed: + +``` +llmgw ctl weights # what could be freed, per node +llmgw ctl weights --collect # delete it +``` + +**Only files llmgw fetched itself are ever candidates.** They are recognisable +because llmgw names them after a hash of their source, so the set still wanted +is computed from the catalog exactly rather than guessed. Anything you put on +a node — by scp, by a shared mount, by hand — is left alone however +unreferenced it looks, and no llmgw command will remove it. The nodes check +every path again before deleting it, so a control plane that is wrong cannot +talk one into losing a model that took a week to copy. + +Which also means llmgw will not reclaim the storage most likely to be filling +your disk. Weights you placed yourself are yours to remove. + +A profile may declare `min_disk_gb`, and a node without that much free is then +not a candidate for it — checked only on nodes that do not already hold the +weights, since those need no room for them. Without the field, disk is not +considered, which is how every profile written before it keeps behaving. + ## When something does not work **The node is `ready` but every replica fails.** Almost always a missing engine