Ilker ManapandClaude Opus 5.5
0416b827fd
The sampler sent the key given for --target, which for the gateway is a client's, so llama.cpp and TensorRT-LLM refused every read and the run reported an idle engine. The engine key now comes from the admin API for the replica at --metrics' address, or --metrics-key; unreadable metrics stop the run before any load, and a step that read none shows why. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1.7 KiB
1.7 KiB
CHANGELOG
SGLang measured, checked and benchmarked as itself
- Fixed: the router read SGLang replicas with vLLM's metric names, and
SGLang was started without
--enable-metrics, so it served no/metricsat all: every scrape failed as though the replica were unreachable, quietly, and requests were routed to it blind. SGLang is now started with--enable-metricsand read in its own names —sglang:num_running_reqs,sglang:num_queue_reqs,sglang:token_usage(the KV pool's fill), the prompt and generation token counters andsglang:num_requests_total, as 0.5.21 publishes them. - Fixed:
llmgw ctl engine-check sglangasked the interpreter for--helpand checked SGLang's flags against Python's. It now askspython -m sglang.launch_server --help, in the environment SGLang was installed into (or--bin). - Fixed: SGLang was given
--tp, which argparse accepts only as an abbreviation of--tp-size; it is given--tp-size. - Fixed:
llmgw ctl bench --metricsread an engine's metrics with the key given for--target— a client's key when the target is the gateway, or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the engine's key, refused every read, and the run reported a queue depth of zero and suggested no thresholds without saying why. The engine key is now asked of the admin API for the replica at--metrics' address (or given with the new--metrics-key); metrics that cannot be read — refused, or in another engine's names than--engine— fail the run before any load, saying why; and a step whose metrics went unread shows-and the reason rather than zeros.