A node's registry holds an adapter for every engine whether or not its binary exists, the agent never said which were installed, and placement carried an Engine field it had never once read. So a vLLM replica was as happily placed on a node with no vLLM: the node registered healthy, the shard failed minutes later with "executable file not found" in the engine's own words, and the reconciler tried again. That is how fifteen gigabytes of safetensors reached this cluster's mirror before anybody noticed it had no card and no vLLM to run them with. Agents now probe what is installed and report it — the protocol has carried engine_versions since it was written and nothing ever filled it in. Placement filters on the key set, `ctl nodes` has a column, and the mirror page's serve warning is a fact instead of a guess about cards, which would have waved through a fleet of eight-GPU nodes that never had vLLM. Probing resolves binaries and runs none of them. The first version asked each engine for --version, which is fine for an engine and not fine for the wrapper scripts people put in front of one: a script that shifts past flags it does not recognise and then execs the server will start an inference server, at agent startup, on every node. The agent's own test suite installs exactly such a script and caught it. So presence is a path lookup, and the version is left to `ctl engine-check`, which is run by somebody who meant to run the engine. SGLang is the exception and has to execute, because a module has no path to look up — but the command is llmgw's own one-line import rather than the engine's launcher. Its exit code is the answer: python3 is on almost every machine, and treating a failed import as "installed" reported SGLang on every node in a fleet that had none, which is the failure being fixed, inverted. Probed once, not per reconnect, and bounded by the context that registration waits on: an agent that cannot reach the control plane is worse than one reporting an incomplete list. A node that reports no engines at all is not excluded anywhere — by placement, or by the page. That is what every agent older than this reports, and emptying a fleet on a control plane upgrade is a worse failure than the one being fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
191 lines
6.6 KiB
Go
191 lines
6.6 KiB
Go
package engines
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"os/exec"
|
|
"regexp"
|
|
"sort"
|
|
"strings"
|
|
"time"
|
|
)
|
|
|
|
// Unknown is the version reported for an engine that is installed and would
|
|
// not say which version it is.
|
|
//
|
|
// A value, not an absent key, because the two mean different things and
|
|
// placement reads the keys: absent is "this node cannot run it", Unknown is
|
|
// "it is here and the probe could not get a number out of it". Collapsing them
|
|
// would make a slow engine look like a missing one and take a node out of
|
|
// scheduling for being slow to answer --version.
|
|
const Unknown = "unknown"
|
|
|
|
// Prober is an engine that can say whether it is installed on this machine.
|
|
//
|
|
// Optional, and asserted for rather than part of Engine: an adapter that has no
|
|
// way to check should report nothing rather than be forced to invent an answer.
|
|
type Prober interface {
|
|
// Probe returns the engine's version, or an error when it is not installed.
|
|
Probe(ctx context.Context) (string, error)
|
|
}
|
|
|
|
// Probe asks every registered engine whether it is here.
|
|
//
|
|
// The result is what the node reports at registration, and its key set is what
|
|
// placement filters on. Until this existed nobody knew: the registry registers
|
|
// every adapter whether or not its binary exists, so a node with no vLLM
|
|
// accepted a vLLM shard, registered healthy, and failed the shard minutes later
|
|
// with "executable file not found" — in the engine's own words, on a machine
|
|
// nobody was watching.
|
|
//
|
|
// Bounded, because it runs on the path to registering and an agent that cannot
|
|
// reach the control plane is worse than one that reports an incomplete list.
|
|
func (r *Registry) Probe(ctx context.Context) map[string]string {
|
|
out := map[string]string{}
|
|
for name, e := range r.engines {
|
|
p, ok := e.(Prober)
|
|
if !ok {
|
|
continue
|
|
}
|
|
v, err := p.Probe(ctx)
|
|
if err != nil {
|
|
continue
|
|
}
|
|
if v == "" {
|
|
v = Unknown
|
|
}
|
|
out[name] = v
|
|
}
|
|
return out
|
|
}
|
|
|
|
// Names of the engines in a probe result, sorted.
|
|
func ProbedNames(m map[string]string) []string {
|
|
out := make([]string, 0, len(m))
|
|
for n := range m {
|
|
out = append(out, n)
|
|
}
|
|
sort.Strings(out)
|
|
return out
|
|
}
|
|
|
|
// probeTimeout bounds one engine's version check.
|
|
//
|
|
// Generous, because vLLM's --version imports torch and that is seconds on a
|
|
// cold page cache, and mean, because this is startup: an engine that has not
|
|
// answered in this long is reported as present-with-unknown-version rather
|
|
// than waited for.
|
|
const probeTimeout = 8 * time.Second
|
|
|
|
// run executes a version command, saying whether the program was even there.
|
|
//
|
|
// The two failures are different answers and the caller has to tell them
|
|
// apart. A binary that is absent is an engine this node cannot run. A binary
|
|
// that is present and exits non-zero is an engine it probably can — unless the
|
|
// command was a test of whether a module imports, where a non-zero exit is the
|
|
// whole answer.
|
|
func run(ctx context.Context, bin string, args ...string) (out string, found bool, ok bool) {
|
|
path, err := exec.LookPath(bin)
|
|
if err != nil {
|
|
return "", false, false
|
|
}
|
|
ctx, cancel := context.WithTimeout(ctx, probeTimeout)
|
|
defer cancel()
|
|
|
|
// Combined, not just stdout: engines print their version to either, and one
|
|
// that chose stderr would otherwise read as having no version at all.
|
|
b, err := exec.CommandContext(ctx, path, args...).CombinedOutput()
|
|
return string(b), true, err == nil
|
|
}
|
|
|
|
// installedBinary reports an engine that is its own executable.
|
|
//
|
|
// It resolves the path and runs nothing. The first version of this ran the
|
|
// engine with --version, which is fine for an engine and not fine for the
|
|
// wrapper scripts people put in front of one: a script that shifts past flags
|
|
// it does not recognise and then execs the server will start an inference
|
|
// server, at agent startup, on every node. The agent's own test suite installs
|
|
// exactly such a script, and caught it.
|
|
//
|
|
// So the version is not read here. Presence is what placement needs, and what
|
|
// the version is for — telling whether an engine still accepts the flags llmgw
|
|
// builds — is `llmgw ctl engine-check`, which is run deliberately by somebody
|
|
// who meant to run the engine.
|
|
func installedBinary(bin string) (string, error) {
|
|
if _, err := exec.LookPath(bin); err != nil {
|
|
return "", errNotInstalled
|
|
}
|
|
return Unknown, nil
|
|
}
|
|
|
|
// versionOfModule probes an engine that is a module inside an interpreter.
|
|
//
|
|
// This one does execute, and safely: the command is llmgw's own one-line
|
|
// import, not the engine's launcher, so there is no wrapper to shift past
|
|
// flags and start a server. There is also no alternative — a module has no
|
|
// path to look up.
|
|
//
|
|
// A non-zero exit is the answer: python3 exists on almost every machine, so
|
|
// presence of the interpreter says nothing. Treating a failed import as
|
|
// "installed" reported SGLang on every node in a fleet that had none, which
|
|
// would take nodes into scheduling for an engine they cannot run — the exact
|
|
// failure this whole change exists to stop.
|
|
func versionOfModule(ctx context.Context, python string, args ...string) (string, error) {
|
|
out, found, ok := run(ctx, python, args...)
|
|
if !found || !ok {
|
|
return "", errNotInstalled
|
|
}
|
|
return firstVersion(out), nil
|
|
}
|
|
|
|
var errNotInstalled = errors.New("not installed on this node")
|
|
|
|
var versionPattern = regexp.MustCompile(`\b\d+\.\d+(\.\d+)?\S*`)
|
|
|
|
// firstVersion is the first thing in the output that looks like a version.
|
|
//
|
|
// Pattern rather than parsing each engine's format: they print "vllm 0.6.3",
|
|
// "version: b4123 (…)", and a paragraph with the number in the middle, and all
|
|
// of those change between releases. What this is for is telling one build from
|
|
// another, which the first number does.
|
|
func firstVersion(s string) string {
|
|
for _, line := range strings.Split(s, "\n") {
|
|
if m := versionPattern.FindString(line); m != "" {
|
|
return strings.TrimRight(m, ",;)")
|
|
}
|
|
}
|
|
return Unknown
|
|
}
|
|
|
|
// Probe reports whether the llama.cpp server is installed.
|
|
func (l *LlamaCpp) Probe(context.Context) (string, error) {
|
|
bin := l.cfg.Bin
|
|
if bin == "" {
|
|
bin = "llama-server"
|
|
}
|
|
return installedBinary(bin)
|
|
}
|
|
|
|
// Probe reports whether vLLM is installed.
|
|
func (v *VLLM) Probe(context.Context) (string, error) {
|
|
bin := v.cfg.Bin
|
|
if bin == "" {
|
|
bin = "vllm"
|
|
}
|
|
return installedBinary(bin)
|
|
}
|
|
|
|
// Probe reports the SGLang version.
|
|
//
|
|
// Through the interpreter, because SGLang is a module rather than a binary:
|
|
// asking the shell for "sglang" finds nothing on a machine that can run it
|
|
// perfectly well.
|
|
func (g *SGLang) Probe(ctx context.Context) (string, error) {
|
|
python := g.cfg.Python
|
|
if python == "" {
|
|
python = "python3"
|
|
}
|
|
return versionOfModule(ctx, python, "-c",
|
|
"import sglang, sys; sys.stdout.write(getattr(sglang, '__version__', ''))")
|
|
}
|