TensorRT-LLM as an engine, and models from Ollama into the mirror
Engine trtllm runs trtllm-serve on one node, through a launcher of llmgw's that demands the replica's key and turns on the Prometheus endpoint its queue and cache gauges are read from; the router reads it at /prometheus/metrics. Brought by the operator (engines.trtllm.python), not installed; checked by ctl engine-check trtllm. Written against the documented CLI, server and metrics and tested with stand-ins: not yet run on a datacenter GPU. mirror import-ollama copies a model's GGUF, and a vision model's projector, from Ollama's directory into the mirror, checked against the digest Ollama named each blob by; mirror import does the same for any local file. The console's Mirror view imports from Ollama with I. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
7940e8593c
28 files changed
+2376
-18
No files matched your search
@@ -28,6 +28,11 @@ type Mapping struct {
|
||||
PromptTokens string
|
||||
GenerationTokens string
|
||||
SuccessTotal string
|
||||
|
||||
// Path is where the engine serves these metrics, when it is not the
|
||||
// poller's configured path: trtllm-serve's /metrics is JSON, and its
|
||||
// Prometheus text is under /prometheus/metrics.
|
||||
Path string
|
||||
}
|
||||
|
||||
// MappingFor returns the metric names for an engine.
|
||||
@@ -41,6 +46,8 @@ func MappingFor(engine string) Mapping {
|
||||
switch engine {
|
||||
case "llamacpp":
|
||||
return LlamaCppMapping()
|
||||
case "trtllm":
|
||||
return TRTLLMMapping()
|
||||
default:
|
||||
return VLLMMapping()
|
||||
}
|
||||
@@ -72,6 +79,33 @@ func LlamaCppMapping() Mapping {
|
||||
}
|
||||
}
|
||||
|
||||
// TRTLLMMapping returns TensorRT-LLM's metric names, as trtllm-serve
|
||||
// publishes them under /prometheus/metrics.
|
||||
//
|
||||
// They are there only when the server runs with return_perf_metrics, and
|
||||
// running and waiting move only with enable_iter_perf_stats; llmgw's launcher
|
||||
// turns both on. The gauges are set from each iteration's statistics, so
|
||||
// they say what the engine was doing when it last iterated: an engine that
|
||||
// has gone idle keeps showing its last batch until the next request moves it.
|
||||
// Running and waiting first appeared in the 1.3 release candidates; an older
|
||||
// server serves none of the three and is reported as breaking the contract,
|
||||
// which is what it does.
|
||||
//
|
||||
// KV cache utilization is used blocks over the pool, a fraction like vLLM's.
|
||||
// The cache's configuration is published under other label names than the
|
||||
// block counts read from vLLM, so it is left unread.
|
||||
func TRTLLMMapping() Mapping {
|
||||
return Mapping{
|
||||
Running: "trtllm_num_requests_running",
|
||||
Waiting: "trtllm_num_requests_waiting",
|
||||
KVCacheUsage: "trtllm_kv_cache_utilization",
|
||||
PromptTokens: "trtllm_prompt_tokens_total",
|
||||
GenerationTokens: "trtllm_generation_tokens_total",
|
||||
SuccessTotal: "trtllm_request_success_total",
|
||||
Path: "/prometheus/metrics",
|
||||
}
|
||||
}
|
||||
|
||||
// VLLMMapping returns vLLM's current metric names.
|
||||
func VLLMMapping() Mapping {
|
||||
return Mapping{
|
||||
|
||||
Reference in new issue
Block a user