Merge engine fixes: SGLang metrics and flags, bench reading guarded metrics

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:20:49 +02:00
commit cc4de36674
12 files changed
+655 -28

No files matched your search

+28
View File
@@ -0,0 +1,28 @@
## CHANGELOG
### SGLang measured, checked and benchmarked as itself
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
SGLang was started without `--enable-metrics`, so it served no `/metrics`
at all: every scrape failed as though the replica were unreachable, quietly,
and requests were routed to it blind. SGLang is now started with
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
prompt and generation token counters and `sglang:num_requests_total`, as
0.5.21 publishes them.
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
`--help` and checked SGLang's flags against Python's. It now asks
`python -m sglang.launch_server --help`, in the environment SGLang was
installed into (or `--bin`).
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
abbreviation of `--tp-size`; it is given `--tp-size`.
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
key given for `--target` — a client's key when the target is the gateway,
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
engine's key, refused every read, and the run reported a queue depth of
zero and suggested no thresholds without saying why. The engine key is now
asked of the admin API for the replica at `--metrics`' address (or given
with the new `--metrics-key`); metrics that cannot be read — refused, or in
another engine's names than `--engine` — fail the run before any load,
saying why; and a step whose metrics went unread shows `-` and the reason
rather than zeros.