diff --git a/CHANGELOG.md b/CHANGELOG.md index 71463a9..c564a46 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,53 @@ is only released once its **exit criterion** is met, not when its code lands. Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period. +## Unreleased + +### Disk, and taking back only what llmgw put there + +Disk was the one resource nothing reported. `statfs` was already being called +on every model directory, but only to ask whether the filesystem was a network +one — the free space in the same struct was discarded, so the only way to find +out why a node was filling up was to ssh to it. + +Nodes now report it, per filesystem rather than per configured directory: free +space belongs to the disk, and two model directories on one disk are one pool +of bytes. Shared filesystems are kept out of a node's own figure, because space +on a filer belongs to everyone mounting it and twenty nodes would otherwise +each report the same free terabyte. It shows in `ctl nodes`, the dashboard and +Prometheus, and a node that reported nothing reads as unknown everywhere — an +older agent must not look like a full disk. + +A profile may declare `min_disk_gb`, and a node without the room is no longer a +candidate, with a reason `explain-placement` prints. Skipped on a node that +already holds the weights: it needs no room for them, and without that the node +best placed to run a model is the one most likely to be rejected. + +`ctl weights` lists model files llmgw downloaded for models no longer in the +catalog, and `--collect` deletes them. The design is mostly about what it will +not do. Only files llmgw fetched are candidates, and that is a calculation +rather than a heuristic — the cache name is a hash of the source, so the wanted +set is computable exactly. Anything an operator put on a node is left alone +however unreferenced it looks. Each node checks every path again before +unlinking it, refusing anything outside its cache directory or without the +cache name, so a control plane that is wrong cannot talk a node into losing a +model that took a week to copy. + +### Fixes + +- **A capability was lost at an indirection.** The controller is handed a + forwarding dispatcher, and reaches past its interface by type assertion to + find what lies beyond scheduling. Deleting weights was implemented on the + agent service and not on the forwarder, so it was simply absent at runtime — + with the whole suite green, because the dispatchers the tests use do not + implement it either and its absence there is the expected case. +- **`ctl tool-check` blamed the engine for what the model did.** It had one + verdict for a failed attempt and told operators not to try another model. + Measured: the engine renders the tool prompt correctly, the coder models do + not use the `` wrapper it asks for, and `qwen2.5-3b-instruct` on + the same engine returns a populated `tool_calls` array. A failed attempt is + now two verdicts, told apart by whether the reply used the wrapper. + ## v0.6.0 — 2026-09-18 Phase 4: a replica can be bigger than one machine, and the fabric it spans diff --git a/docs/install.md b/docs/install.md index 4819b78..3b1b03d 100644 --- a/docs/install.md +++ b/docs/install.md @@ -595,6 +595,38 @@ fleet means the measurement is missing, not that the fleet is idle. And `llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown, which is what a binary built without the nvml tag reports. +## Reclaiming disk + +`llmgw ctl nodes` has a DISK column: free space against the size it is free +out of, for the filesystems holding model weights. A dash means the node was +not asked — an agent with no `model_dirs`, or one older than this — and not +that it has no space. Shared filesystems are excluded from a node's figure, +because space on a filer belongs to everyone mounting it. + +Weights llmgw downloaded for models that are no longer in the catalog can be +listed and removed: + +``` +llmgw ctl weights # what could be freed, per node +llmgw ctl weights --collect # delete it +``` + +**Only files llmgw fetched itself are ever candidates.** They are recognisable +because llmgw names them after a hash of their source, so the set still wanted +is computed from the catalog exactly rather than guessed. Anything you put on +a node — by scp, by a shared mount, by hand — is left alone however +unreferenced it looks, and no llmgw command will remove it. The nodes check +every path again before deleting it, so a control plane that is wrong cannot +talk one into losing a model that took a week to copy. + +Which also means llmgw will not reclaim the storage most likely to be filling +your disk. Weights you placed yourself are yours to remove. + +A profile may declare `min_disk_gb`, and a node without that much free is then +not a candidate for it — checked only on nodes that do not already hold the +weights, since those need no room for them. Without the field, disk is not +considered, which is how every profile written before it keeps behaving. + ## When something does not work **The node is `ready` but every replica fails.** Almost always a missing engine