Stop placing engines on nodes that do not have them
CI / test (push) Successful in 1m42s

A node's registry holds an adapter for every engine whether or not its binary
exists, the agent never said which were installed, and placement carried an
Engine field it had never once read. So a vLLM replica was as happily placed on
a node with no vLLM: the node registered healthy, the shard failed minutes
later with "executable file not found" in the engine's own words, and the
reconciler tried again.

That is how fifteen gigabytes of safetensors reached this cluster's mirror
before anybody noticed it had no card and no vLLM to run them with.

Agents now probe what is installed and report it — the protocol has carried
engine_versions since it was written and nothing ever filled it in. Placement
filters on the key set, `ctl nodes` has a column, and the mirror page's serve
warning is a fact instead of a guess about cards, which would have waved
through a fleet of eight-GPU nodes that never had vLLM.

Probing resolves binaries and runs none of them. The first version asked each
engine for --version, which is fine for an engine and not fine for the wrapper
scripts people put in front of one: a script that shifts past flags it does not
recognise and then execs the server will start an inference server, at agent
startup, on every node. The agent's own test suite installs exactly such a
script and caught it. So presence is a path lookup, and the version is left to
`ctl engine-check`, which is run by somebody who meant to run the engine.

SGLang is the exception and has to execute, because a module has no path to
look up — but the command is llmgw's own one-line import rather than the
engine's launcher. Its exit code is the answer: python3 is on almost every
machine, and treating a failed import as "installed" reported SGLang on every
node in a fleet that had none, which is the failure being fixed, inverted.

Probed once, not per reconnect, and bounded by the context that registration
waits on: an agent that cannot reach the control plane is worse than one
reporting an incomplete list.

A node that reports no engines at all is not excluded anywhere — by placement,
or by the page. That is what every agent older than this reports, and emptying
a fleet on a control plane upgrade is a worse failure than the one being fixed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 16:11:55 +02:00
1 parent e034862b79
commit 2acd794022
9 files changed
+500 -21

No files matched your search

+10
View File
@@ -73,6 +73,12 @@ type NodeView struct {
// what decides whether a CPU profile can be placed here, so it belongs
// where someone looks when a model will not schedule.
MemoryGB int `json:"memory_gb,omitempty"`
// Engines are the inference engines installed here, as the agent found
// them. Absent means the agent is older than the probe and said nothing,
// which is not the same as having none — placement treats it that way too,
// or a control plane upgrade would empty a fleet.
Engines []string `json:"engines,omitempty"`
// Models are the weights this node can see. Absent means it was not asked
// to look, which is not the same as holding none.
Models []string `json:"models_on_disk,omitempty"`
@@ -211,6 +217,10 @@ func (a *API) Nodes() []NodeView {
v.DiskFreeGB = int(free / (1 << 30))
v.DiskTotalGB = int(total / (1 << 30))
}
for name := range n.EngineVersions {
v.Engines = append(v.Engines, name)
}
slices.Sort(v.Engines)
for _, w := range n.Weights {
v.Models = append(v.Models, w.Path)
}