TensorRT-LLM as an engine, and models from Ollama into the mirror

Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.

mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:02:01 +02:00
1 parent 30f7f3474a
commit 7940e8593c
28 files changed
+2376 -18

No files matched your search

+26
View File
@@ -264,6 +264,7 @@ rather than repeated.
| `models` | `[]catalog.Model` | [Server](#server) |
| `endpoints` | `[]Endpoint` | [Server](#server) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
| `store` | `string` | [Threads](#threads) |
| `retain_hours` | `int` | [Threads](#threads) |
| `dial` | `time.Duration` | [Timeouts](#timeouts) |
@@ -331,6 +332,7 @@ rather than repeated.
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
@@ -2175,6 +2177,21 @@ against sim-engines possible without a control plane.
Bin is the llmgw binary itself; the engine is one of its subcommands.
Empty means whatever started this agent, which is right almost always.
## TRTLLMConfig
### `python`
`string`
Python is the interpreter of the environment TensorRT-LLM is installed
in: <venv>/bin/python for a virtual environment, python3 in NVIDIA's
container. Naming it is what makes this node offer the engine; empty
offers nothing.
The interpreter rather than trtllm-serve, because llmgw starts the
server through a few lines of its own that make it demand the
gateway's key, which trtllm-serve has no option for.
## Threads
### `store`
@@ -2633,6 +2650,15 @@ Takes a [SGLangConfig](#sglangconfig) block.
Takes a [LlamaCppConfig](#llamacppconfig) block.
### `trtllm`
`TRTLLMConfig`
TRTLLM is TensorRT-LLM, which the agent does not install: a node
offers it only when its interpreter is named here.
Takes a [TRTLLMConfig](#trtllmconfig) block.
### `sim`
`SimConfig`