The router read SGLang replicas with vLLM's names, and SGLang was started
without --enable-metrics, so it served no /metrics: every scrape failed as
if unreachable and the replica was routed to blind. It now gets
--enable-metrics and a mapping of its own (sglang:num_running_reqs,
num_queue_reqs, token_usage, the token counters and num_requests_total, as
0.5.21 publishes them). engine-check asks python -m sglang.launch_server
--help instead of Python's own help, and SGLang is given --tp-size rather
than the --tp argparse only takes as an abbreviation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.
mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A release was a git tag. Installing llmgw on a node meant having a Go
toolchain, and the onboarding script copied a binary from wherever it was run —
which works and quietly makes whoever runs it the build server, with the fleet
ending up on whatever was in their working tree. Tags now build static
linux/amd64, linux/arm64 and darwin binaries and attach them. CGO_ENABLED=0 is
the point: it has to run on a node whose libc is older than the builder's, and
llmgw has no C dependency because modernc.org/sqlite is pure Go precisely so
this stays true. Verified: "not a dynamic executable", runs on a node, reports
its version.
`ctl engine-check` closes the other half of the phase 4 gap. llmgw's launch
arguments are read from each engine's documentation, and engines rename flags
between versions; when that happens the failure arrives minutes into a model
load, in the engine's own error message, on a node nobody was watching. It
builds the widest command llmgw would ever produce — multi-node, since a flag
that only appears there is the one nobody notices has gone — and checks each
flag against the engine's own --help. No GPU: a CLI prints its help anywhere.
Verified against llama.cpp b10991, all seven flags accepted.
Flags are matched on a word boundary, so --port is not satisfied by
--port-range. Without that, a renamed flag would sail through the check that
exists to catch renamed flags.
It also warns when the help text looks truncated, which is how the first run
went: llama.cpp's help cut to 400 of its 725 lines reported five flags as
missing that the running cluster uses.
TestSidecarRunsBeforeEngine was flaky under a full run, and worse than flaky —
its wait loop fell through on timeout, so a slow machine turned the ordering
assertion into a no-op and the failure surfaced later as something else. It now
asserts, with a deadline that suits a machine running every other package at
the same time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
