Merge engine fixes: SGLang metrics and flags, bench reading guarded metrics
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
cc4de36674
12 files changed
+655
-28
No files matched your search
@@ -0,0 +1,28 @@
|
||||
## CHANGELOG
|
||||
|
||||
### SGLang measured, checked and benchmarked as itself
|
||||
|
||||
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
|
||||
SGLang was started without `--enable-metrics`, so it served no `/metrics`
|
||||
at all: every scrape failed as though the replica were unreachable, quietly,
|
||||
and requests were routed to it blind. SGLang is now started with
|
||||
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
|
||||
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
|
||||
prompt and generation token counters and `sglang:num_requests_total`, as
|
||||
0.5.21 publishes them.
|
||||
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
|
||||
`--help` and checked SGLang's flags against Python's. It now asks
|
||||
`python -m sglang.launch_server --help`, in the environment SGLang was
|
||||
installed into (or `--bin`).
|
||||
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
|
||||
abbreviation of `--tp-size`; it is given `--tp-size`.
|
||||
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
|
||||
key given for `--target` — a client's key when the target is the gateway,
|
||||
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
|
||||
engine's key, refused every read, and the run reported a queue depth of
|
||||
zero and suggested no thresholds without saying why. The engine key is now
|
||||
asked of the admin API for the replica at `--metrics`' address (or given
|
||||
with the new `--metrics-key`); metrics that cannot be read — refused, or in
|
||||
another engine's names than `--engine` — fail the run before any load,
|
||||
saying why; and a step whose metrics went unread shows `-` and the reason
|
||||
rather than zeros.
|
||||
Reference in new issue
Block a user