SGLang read in its own metric names, started serving them, and asked for its own help

The router read SGLang replicas with vLLM's names, and SGLang was started
without --enable-metrics, so it served no /metrics: every scrape failed as
if unreachable and the replica was routed to blind. It now gets
--enable-metrics and a mapping of its own (sglang:num_running_reqs,
num_queue_reqs, token_usage, the token counters and num_requests_total, as
0.5.21 publishes them). engine-check asks python -m sglang.launch_server
--help instead of Python's own help, and SGLang is given --tp-size rather
than the --tp argparse only takes as an abbreviation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:20:25 +02:00
1 parent d41104a452
commit 8cc8d7139b
6 files changed
+251 -3

No files matched your search

+35
View File
@@ -48,6 +48,8 @@ func MappingFor(engine string) Mapping {
return LlamaCppMapping()
case "trtllm":
return TRTLLMMapping()
case "sglang":
return SGLangMapping()
default:
return VLLMMapping()
}
@@ -106,6 +108,39 @@ func TRTLLMMapping() Mapping {
}
}
// SGLangMapping returns SGLang's metric names, as sglang.launch_server
// publishes them under /metrics (checked against 0.5.21's collector).
//
// They are there only when the server runs with --enable-metrics, which
// llmgw's launch command passes; without it /metrics is not served at all.
// SGLang exempts /metrics from its API key, so the key the poller sends is
// ignored rather than needed.
//
// "Queue" is SGLang's word for vLLM's waiting. The gauges are set by the
// scheduler as it reports its batches, and at least every thirty seconds
// while it is idle, so a server that has only just started may serve none of
// them yet and is reported incomplete until it does.
//
// token_usage is the fraction of the KV pool in use, as vLLM's
// kv_cache_usage_perc is (for a model with sliding-window layers, the fuller
// of its two pools). cache_hit_rate is a different number — how much of the
// prompts the prefix cache answered — and is not what saturation means.
//
// num_requests_total counts the requests that finished, the nearest thing it
// has to vLLM's success counter; like the token counters it is split by
// is_streaming and summed. There is no cache configuration to read block
// sizes from.
func SGLangMapping() Mapping {
return Mapping{
Running: "sglang:num_running_reqs",
Waiting: "sglang:num_queue_reqs",
KVCacheUsage: "sglang:token_usage",
PromptTokens: "sglang:prompt_tokens_total",
GenerationTokens: "sglang:generation_tokens_total",
SuccessTotal: "sglang:num_requests_total",
}
}
// VLLMMapping returns vLLM's current metric names.
func VLLMMapping() Mapping {
return Mapping{