Ship binaries with releases, and ask engines whether our flags still exist

A release was a git tag. Installing llmgw on a node meant having a Go
toolchain, and the onboarding script copied a binary from wherever it was run —
which works and quietly makes whoever runs it the build server, with the fleet
ending up on whatever was in their working tree. Tags now build static
linux/amd64, linux/arm64 and darwin binaries and attach them. CGO_ENABLED=0 is
the point: it has to run on a node whose libc is older than the builder's, and
llmgw has no C dependency because modernc.org/sqlite is pure Go precisely so
this stays true. Verified: "not a dynamic executable", runs on a node, reports
its version.

`ctl engine-check` closes the other half of the phase 4 gap. llmgw's launch
arguments are read from each engine's documentation, and engines rename flags
between versions; when that happens the failure arrives minutes into a model
load, in the engine's own error message, on a node nobody was watching. It
builds the widest command llmgw would ever produce — multi-node, since a flag
that only appears there is the one nobody notices has gone — and checks each
flag against the engine's own --help. No GPU: a CLI prints its help anywhere.
Verified against llama.cpp b10991, all seven flags accepted.

Flags are matched on a word boundary, so --port is not satisfied by
--port-range. Without that, a renamed flag would sail through the check that
exists to catch renamed flags.

It also warns when the help text looks truncated, which is how the first run
went: llama.cpp's help cut to 400 of its 725 lines reported five flags as
missing that the running cluster uses.

TestSidecarRunsBeforeEngine was flaky under a full run, and worse than flaky —
its wait loop fell through on timeout, so a slow machine turned the ordering
assertion into a no-op and the failure surfaced later as something else. It now
asserts, with a deadline that suits a machine running every other package at
the same time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 16:44:29 +02:00
1 parent 7b4acd9284
commit 5b425c32e5
8 files changed
+407 -13

No files matched your search

+21 -9
View File
@@ -557,15 +557,27 @@ func TestSidecarRunsBeforeEngine(t *testing.T) {
t.Fatal(err)
}
deadline := time.Now().Add(3 * time.Second)
for time.Now().Before(deadline) {
if b, err := os.ReadFile(order); err == nil && strings.Count(string(b), "\n") == 2 {
if got := strings.Fields(string(b)); got[0] != "sidecar" || got[1] != "engine" {
t.Fatalf("start order was %v", got)
}
break
// Generous, because this waits on two processes starting under the race
// detector while every other package is competing for the same cores. Three
// seconds was enough alone and not in a full run.
var started []string
deadline := time.Now().Add(20 * time.Second)
for time.Now().Before(deadline) && len(started) < 2 {
if b, err := os.ReadFile(order); err == nil {
started = strings.Fields(string(b))
}
time.Sleep(20 * time.Millisecond)
if len(started) < 2 {
time.Sleep(20 * time.Millisecond)
}
}
// Asserted rather than broken out of: the original fell through on timeout,
// so a slow machine turned the ordering check into a no-op and the failure
// surfaced later as something else.
if len(started) != 2 {
t.Fatalf("only %d of 2 processes started: %v", len(started), started)
}
if started[0] != "sidecar" || started[1] != "engine" {
t.Fatalf("start order was %v", started)
}
// The shard's pid is the engine's, not the sidecar's: the engine's exit is
@@ -574,7 +586,7 @@ func TestSidecarRunsBeforeEngine(t *testing.T) {
t.Fatal("no engine pid recorded")
}
// Both processes appear in one log tail, distinguishable by prefix.
deadline = time.Now().Add(3 * time.Second)
deadline = time.Now().Add(20 * time.Second)
for time.Now().Before(deadline) {
logs := strings.Join(sh.Logs(20), "\n")
if strings.Contains(logs, "[sidecar] hello-from-sidecar") &&