SGLang read in its own metric names, started serving them, and asked for its own help
The router read SGLang replicas with vLLM's names, and SGLang was started without --enable-metrics, so it served no /metrics: every scrape failed as if unreachable and the replica was routed to blind. It now gets --enable-metrics and a mapping of its own (sglang:num_running_reqs, num_queue_reqs, token_usage, the token counters and num_requests_total, as 0.5.21 publishes them). engine-check asks python -m sglang.launch_server --help instead of Python's own help, and SGLang is given --tp-size rather than the --tp argparse only takes as an abbreviation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
8cc8d7139b
6 files changed
+251
-3
No files matched your search
@@ -48,6 +48,8 @@ func MappingFor(engine string) Mapping {
|
||||
return LlamaCppMapping()
|
||||
case "trtllm":
|
||||
return TRTLLMMapping()
|
||||
case "sglang":
|
||||
return SGLangMapping()
|
||||
default:
|
||||
return VLLMMapping()
|
||||
}
|
||||
@@ -106,6 +108,39 @@ func TRTLLMMapping() Mapping {
|
||||
}
|
||||
}
|
||||
|
||||
// SGLangMapping returns SGLang's metric names, as sglang.launch_server
|
||||
// publishes them under /metrics (checked against 0.5.21's collector).
|
||||
//
|
||||
// They are there only when the server runs with --enable-metrics, which
|
||||
// llmgw's launch command passes; without it /metrics is not served at all.
|
||||
// SGLang exempts /metrics from its API key, so the key the poller sends is
|
||||
// ignored rather than needed.
|
||||
//
|
||||
// "Queue" is SGLang's word for vLLM's waiting. The gauges are set by the
|
||||
// scheduler as it reports its batches, and at least every thirty seconds
|
||||
// while it is idle, so a server that has only just started may serve none of
|
||||
// them yet and is reported incomplete until it does.
|
||||
//
|
||||
// token_usage is the fraction of the KV pool in use, as vLLM's
|
||||
// kv_cache_usage_perc is (for a model with sliding-window layers, the fuller
|
||||
// of its two pools). cache_hit_rate is a different number — how much of the
|
||||
// prompts the prefix cache answered — and is not what saturation means.
|
||||
//
|
||||
// num_requests_total counts the requests that finished, the nearest thing it
|
||||
// has to vLLM's success counter; like the token counters it is split by
|
||||
// is_streaming and summed. There is no cache configuration to read block
|
||||
// sizes from.
|
||||
func SGLangMapping() Mapping {
|
||||
return Mapping{
|
||||
Running: "sglang:num_running_reqs",
|
||||
Waiting: "sglang:num_queue_reqs",
|
||||
KVCacheUsage: "sglang:token_usage",
|
||||
PromptTokens: "sglang:prompt_tokens_total",
|
||||
GenerationTokens: "sglang:generation_tokens_total",
|
||||
SuccessTotal: "sglang:num_requests_total",
|
||||
}
|
||||
}
|
||||
|
||||
// VLLMMapping returns vLLM's current metric names.
|
||||
func VLLMMapping() Mapping {
|
||||
return Mapping{
|
||||
|
||||
Reference in new issue
Block a user