Refresh disk and weights while the agent runs, not once when it starts

Found by answering "where can we see this live". `ctl weights` was offering a
file for deletion that I had deleted two days earlier, and the node's disk
figure was the one it had when its agent started. Discover() is called at
registration and nowhere else, so both were frozen.

Disk is a fact about a moment, not about a machine. Sampled once, the figure
is wrong on exactly the node worth watching — the one busy filling up — and it
stays wrong until that agent restarts. The heartbeat already refreshes GPU
memory for the same reason, with a comment about noticing a foreign process
"after registration"; disk belonged beside it and I did not put it there.

The stale list cut both ways. A model fetched after registration was invisible
until a restart, so placement could not prefer the node that had just spent ten
minutes downloading it, and the disk check could not see the space it used.

Storage goes in every heartbeat, being a few numbers per filesystem. The file
list goes only when a fingerprint of paths and sizes changes, with a flag
saying the list is meant — without that, an empty list would be ambiguous
between "this node now holds nothing" and "nothing to report", and the second
reading would empty a node's inventory five seconds after it registered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-20 09:57:25 +02:00
1 parent cf20570ff1
commit 45e913bd10
5 files changed
+261 -15

No files matched your search

+30 -1
View File
@@ -14,6 +14,7 @@ import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1/llmgwv1connect"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
@@ -239,7 +240,8 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
used := hb.GetGpuMemoryUsedMb()
problems := hb.GetNewProblems()
if len(used) == 0 && len(problems) == 0 {
storage := hb.GetStorage()
if len(used) == 0 && len(problems) == 0 && len(storage) == 0 && !hb.GetWeightsChanged() {
return
}
@@ -259,6 +261,33 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
// conservative direction.
n.GPUs[0].Health.Problems = append(n.GPUs[0].Health.Problems, problems...)
}
// Disk is a fact about a moment, not about a machine. Sampled only at
// registration it is wrong on exactly the node worth watching — the one
// busy filling up — and it stays wrong until that agent restarts.
if len(storage) > 0 {
n.Storage = n.Storage[:0]
for _, st := range storage {
n.Storage = append(n.Storage, discovery.Storage{
Path: st.GetPath(),
FreeBytes: st.GetFreeBytes(),
TotalBytes: st.GetTotalBytes(),
Shared: st.GetShared(),
})
}
}
// Only when the agent says the list is meant. An empty list without the
// flag is silence, and treating it as "this node holds nothing" would
// empty a node's inventory on every heartbeat.
if hb.GetWeightsChanged() {
n.Weights = n.Weights[:0]
for _, w := range hb.GetWeights() {
n.Weights = append(n.Weights, discovery.LocalWeights{
Path: w.GetPath(), SizeBytes: w.GetSizeBytes(), Shared: w.GetShared(),
})
}
}
s.inv.Upsert(n)
}