Refresh disk and weights while the agent runs, not once when it starts
CI / test (push) Successful in 1m46s
CI / test (push) Successful in 1m46s
Found by answering "where can we see this live". `ctl weights` was offering a file for deletion that I had deleted two days earlier, and the node's disk figure was the one it had when its agent started. Discover() is called at registration and nowhere else, so both were frozen. Disk is a fact about a moment, not about a machine. Sampled once, the figure is wrong on exactly the node worth watching — the one busy filling up — and it stays wrong until that agent restarts. The heartbeat already refreshes GPU memory for the same reason, with a comment about noticing a foreign process "after registration"; disk belonged beside it and I did not put it there. The stale list cut both ways. A model fetched after registration was invisible until a restart, so placement could not prefer the node that had just spent ten minutes downloading it, and the disk check could not see the space it used. Storage goes in every heartbeat, being a few numbers per filesystem. The file list goes only when a fingerprint of paths and sizes changes, with a flag saying the list is meant — without that, an empty list would be ambiguous between "this node now holds nothing" and "nothing to report", and the second reading would empty a node's inventory five seconds after it registered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
cf20570ff1
commit
45e913bd10
5 files changed
+261
-15
No files matched your search
@@ -3,7 +3,9 @@ package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -172,6 +174,11 @@ type Agent struct {
|
||||
// report can name both. The control plane keys on the replica.
|
||||
shardReplica map[string]string
|
||||
fetcher *weights.Fetcher
|
||||
|
||||
// lastWeights is the fingerprint of the file list most recently sent, so a
|
||||
// heartbeat carries the list only when it changed. Touched only by the
|
||||
// heartbeat goroutine, which is the only thing that reads or writes it.
|
||||
lastWeights string
|
||||
}
|
||||
|
||||
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
|
||||
@@ -580,9 +587,49 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
|
||||
for _, g := range inv.GPUs {
|
||||
hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
|
||||
}
|
||||
|
||||
// Rescanned here rather than reused from registration. Disk is not a fact
|
||||
// about a machine, it is a fact about a moment: a node that had 200 GB when
|
||||
// its agent started can be full an hour later, and it is precisely the node
|
||||
// busy filling up whose figure matters.
|
||||
for _, st := range discovery.ScanStorage(a.cfg.ModelDirs) {
|
||||
hb.Storage = append(hb.Storage, &llmgwv1.Storage{
|
||||
Path: st.Path, FreeBytes: st.FreeBytes,
|
||||
TotalBytes: st.TotalBytes, Shared: st.Shared,
|
||||
})
|
||||
}
|
||||
|
||||
// The file list is longer and changes rarely, so it goes only when it has.
|
||||
// The flag is what makes an empty list mean "this node now holds nothing"
|
||||
// rather than "nothing to report" — without it, the first heartbeat after
|
||||
// a model was deleted would be indistinguishable from silence.
|
||||
weights := discovery.ScanWeights(a.cfg.ModelDirs)
|
||||
if fp := weightsFingerprint(weights); fp != a.lastWeights {
|
||||
a.lastWeights = fp
|
||||
hb.WeightsChanged = true
|
||||
for _, w := range weights {
|
||||
hb.Weights = append(hb.Weights, &llmgwv1.LocalWeights{
|
||||
Path: w.Path, SizeBytes: w.SizeBytes, Shared: w.Shared,
|
||||
})
|
||||
}
|
||||
}
|
||||
return &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Heartbeat{Heartbeat: hb}}
|
||||
}
|
||||
|
||||
// weightsFingerprint is enough to notice a model appearing, disappearing or
|
||||
// finishing its download.
|
||||
//
|
||||
// Paths and sizes, because a fetch in progress has the same name as the file
|
||||
// it will become — under a different name until it is renamed, but a shared
|
||||
// filesystem can also grow a file in place.
|
||||
func weightsFingerprint(ws []discovery.LocalWeights) string {
|
||||
h := sha256.New()
|
||||
for _, w := range ws {
|
||||
fmt.Fprintf(h, "%s:%d\n", w.Path, w.SizeBytes)
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
|
||||
// removeWeights deletes cached model files the control plane says are no
|
||||
// longer needed.
|
||||
//
|
||||
|
||||
Reference in new issue
Block a user