Carry a download's progress to where somebody is looking at it
CI / test (push) Successful in 1m28s

The agent has always logged a percentage as it fetches weights, with a comment
saying a replica pulling a hundred gigabytes and one that is wedged look
identical from the control plane. They still did: the log is on the node, and
the node is not where anyone is looking when a replica has been starting for
ten minutes.

Progress now rides on the shard status and appears under the replica in `ctl
replicas` — a line rather than a column, because most replicas are not
downloading anything and a column would be empty for all of them.

It is deliberately not a state transition. Treating each chunk as one would
re-mark the shard as starting on every report and reset the timer that decides
when a slow start has become a failed one, so an hour-long download would keep
a doomed replica alive indefinitely. It arrives on its own method for that
reason, and a real transition clears it — otherwise a finished download sits in
the listing at ninety-nine percent under a shard that has been serving for an
hour.

Watchers are keyed by source rather than by shard, because a fetch is
single-flight: eight shards starting together produce one download and all
eight are waiting on it.

No percentage is invented when the server sent no length. One that stops moving
is worse than a byte count that keeps rising, so the field is absent and the
listing says "so far".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 07:49:08 +02:00
1 parent 100f10ac17
commit a35e8796d2
11 files changed
+447 -60

No files matched your search

+27 -2
View File
@@ -131,6 +131,22 @@ type ShardView struct {
GPUs []int `json:"gpus"`
State string `json:"state"`
Err string `json:"error,omitempty"`
// Fetch is the download this shard is waiting on, absent when it is not
// waiting on one. It is what tells a replica pulling a hundred gigabytes
// apart from one that is wedged — which look identical without it.
Fetch *FetchView `json:"fetch,omitempty"`
}
// FetchView is a download in flight.
type FetchView struct {
Source string `json:"source"`
BytesDone int64 `json:"bytes_done"`
// BytesTotal is 0 when the server did not say how large the file is.
BytesTotal int64 `json:"bytes_total,omitempty"`
// Percent is absent rather than zero when the total is unknown: an
// invented percentage that stops moving is worse than a byte count that
// keeps rising.
Percent *int `json:"percent,omitempty"`
}
// ExplainView is why a model could not be placed.
@@ -262,10 +278,19 @@ func (a *API) Replicas(model string) []ReplicaView {
v.IdleSec = int64(time.Since(rep.LastUsedAt).Seconds())
}
for _, s := range rep.Shards {
v.Shards = append(v.Shards, ShardView{
sv := ShardView{
ID: s.ID, Node: s.NodeID, Rank: s.Rank,
GPUs: s.GPUIndices, State: s.State.String(), Err: s.Err,
})
}
if f := s.Fetch; f != nil {
sv.Fetch = &FetchView{
Source: f.Source, BytesDone: f.BytesDone, BytesTotal: f.BytesTotal,
}
if p, ok := f.Percent(); ok {
sv.Fetch.Percent = &p
}
}
v.Shards = append(v.Shards, sv)
}
out = append(out, v)
}