Ilker ManapandClaude Opus 5.5
0416b827fd
The sampler sent the key given for --target, which for the gateway is a client's, so llama.cpp and TensorRT-LLM refused every read and the run reported an idle engine. The engine key now comes from the admin API for the replica at --metrics' address, or --metrics-key; unreadable metrics stop the run before any load, and a step that read none shows why. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
29 lines
1.7 KiB
Markdown
29 lines
1.7 KiB
Markdown
## CHANGELOG
|
|
|
|
### SGLang measured, checked and benchmarked as itself
|
|
|
|
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
|
|
SGLang was started without `--enable-metrics`, so it served no `/metrics`
|
|
at all: every scrape failed as though the replica were unreachable, quietly,
|
|
and requests were routed to it blind. SGLang is now started with
|
|
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
|
|
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
|
|
prompt and generation token counters and `sglang:num_requests_total`, as
|
|
0.5.21 publishes them.
|
|
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
|
|
`--help` and checked SGLang's flags against Python's. It now asks
|
|
`python -m sglang.launch_server --help`, in the environment SGLang was
|
|
installed into (or `--bin`).
|
|
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
|
|
abbreviation of `--tp-size`; it is given `--tp-size`.
|
|
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
|
|
key given for `--target` — a client's key when the target is the gateway,
|
|
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
|
|
engine's key, refused every read, and the run reported a queue depth of
|
|
zero and suggested no thresholds without saying why. The engine key is now
|
|
asked of the admin API for the replica at `--metrics`' address (or given
|
|
with the new `--metrics-key`); metrics that cannot be read — refused, or in
|
|
another engine's names than `--engine` — fail the run before any load,
|
|
saying why; and a step whose metrics went unread shows `-` and the reason
|
|
rather than zeros.
|