TensorRT-LLM as an engine, and models from Ollama into the mirror

Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.

mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:02:01 +02:00
1 parent 30f7f3474a
commit 7940e8593c
28 files changed
+2376 -18

No files matched your search

+34
View File
@@ -28,6 +28,11 @@ type Mapping struct {
PromptTokens string
GenerationTokens string
SuccessTotal string
// Path is where the engine serves these metrics, when it is not the
// poller's configured path: trtllm-serve's /metrics is JSON, and its
// Prometheus text is under /prometheus/metrics.
Path string
}
// MappingFor returns the metric names for an engine.
@@ -41,6 +46,8 @@ func MappingFor(engine string) Mapping {
switch engine {
case "llamacpp":
return LlamaCppMapping()
case "trtllm":
return TRTLLMMapping()
default:
return VLLMMapping()
}
@@ -72,6 +79,33 @@ func LlamaCppMapping() Mapping {
}
}
// TRTLLMMapping returns TensorRT-LLM's metric names, as trtllm-serve
// publishes them under /prometheus/metrics.
//
// They are there only when the server runs with return_perf_metrics, and
// running and waiting move only with enable_iter_perf_stats; llmgw's launcher
// turns both on. The gauges are set from each iteration's statistics, so
// they say what the engine was doing when it last iterated: an engine that
// has gone idle keeps showing its last batch until the next request moves it.
// Running and waiting first appeared in the 1.3 release candidates; an older
// server serves none of the three and is reported as breaking the contract,
// which is what it does.
//
// KV cache utilization is used blocks over the pool, a fraction like vLLM's.
// The cache's configuration is published under other label names than the
// block counts read from vLLM, so it is left unread.
func TRTLLMMapping() Mapping {
return Mapping{
Running: "trtllm_num_requests_running",
Waiting: "trtllm_num_requests_waiting",
KVCacheUsage: "trtllm_kv_cache_utilization",
PromptTokens: "trtllm_prompt_tokens_total",
GenerationTokens: "trtllm_generation_tokens_total",
SuccessTotal: "trtllm_request_success_total",
Path: "/prometheus/metrics",
}
}
// VLLMMapping returns vLLM's current metric names.
func VLLMMapping() Mapping {
return Mapping{