The agent has always logged a percentage as it fetches weights, with a comment
saying a replica pulling a hundred gigabytes and one that is wedged look
identical from the control plane. They still did: the log is on the node, and
the node is not where anyone is looking when a replica has been starting for
ten minutes.
Progress now rides on the shard status and appears under the replica in `ctl
replicas` — a line rather than a column, because most replicas are not
downloading anything and a column would be empty for all of them.
It is deliberately not a state transition. Treating each chunk as one would
re-mark the shard as starting on every report and reset the timer that decides
when a slow start has become a failed one, so an hour-long download would keep
a doomed replica alive indefinitely. It arrives on its own method for that
reason, and a real transition clears it — otherwise a finished download sits in
the listing at ninety-nine percent under a shard that has been serving for an
hour.
Watchers are keyed by source rather than by shard, because a fetch is
single-flight: eight shards starting together produce one download and all
eight are waiting on it.
No percentage is invented when the server sent no length. One that stops moving
is worse than a byte count that keeps rising, so the field is absent and the
listing says "so far".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two halves of the same problem: nothing stopped a model being placed on a node
that could not hold it, and nothing ever removed the weights of a model that
had been deleted from the catalog.
Placement now has a disk check, shaped exactly like the memory one it sits
beside, with one addition: it is skipped on a node that already holds the
weights. A node holding the file needs no room for it, and without that the
node best placed to run a model — because it already has it — is the one most
likely to be rejected, since holding the file is what used the space.
Reclaiming is the dangerous half, so the whole design is about what it will
not do. llmgw deletes only files it fetched itself, and that is a calculation
rather than a heuristic: the cache name is a hash of the source, so the set of
wanted files can be computed from the catalog exactly. Anything without that
name was put there by an operator — scp, a shared mount, a copy by hand — and
is none of this system's business however unreferenced it looks. A model that
took a week to copy must not be removable by a command someone ran to tidy up.
The node checks every path again before unlinking it, and that duplication is
the point. The control plane knows the catalog; the node knows its disk. A
control plane that is wrong, newer than the agent, or tampered with must not be
able to talk a node into deleting something, so the agent refuses anything
outside its own cache directory or without the cache name — including a path
that tries to traverse out of it.
Listing is the default and deleting takes --collect. The summary says "asked to
remove", because each node decides, and one that refuses a path it cannot prove
it fetched is this working rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
