TensorRT-LLM as an engine, and models from Ollama into the mirror

Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.

mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:02:01 +02:00
1 parent 30f7f3474a
commit 7940e8593c
28 files changed
+2376 -18

No files matched your search

+119 -2
View File
@@ -383,6 +383,46 @@ llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
or from the admin page, or by copying files into the directory by hand — the
mirror is a directory, and anything that puts files in it works.
Files already on the mirror machine go in with `import`, which copies them the
way a download is kept — under a temporary name until whole, the checksum
beside them for nodes to check against, a version never replaced — and checks
that a `.gguf` is one:
```
llmgw mirror import ./qwen2.5-7b-instruct-q4_k_m.gguf --model qwen2.5-7b-instruct \
--version 2026-10 --sha256 <the checksum its publisher gave>
```
#### From Ollama
Models you already run in Ollama are GGUFs llama.cpp serves as they are.
On the mirror machine, with Ollama's models readable there:
```
llmgw mirror import-ollama # lists what Ollama has
llmgw mirror import-ollama llama3.2:3b
```
It finds Ollama's models directory where Ollama keeps it — `OLLAMA_MODELS`,
`~/.ollama/models`, or `/usr/share/ollama/.ollama/models` for Ollama's Linux
service, which needs `sudo` to read — or takes `--ollama-dir`. It copies the
model's GGUF, and a vision model's projector beside it, each checked against
the sha256 Ollama named its file by; a file that does not match is not
mirrored, and nothing of the model is. The version is the tag and the start of
the digest (`llama3.2/3b-a80c4f17acd5/llama3.2-3b.gguf`), because a tag like
`latest` moves in Ollama and a mirror version must not; `--model` and
`--version` choose otherwise. It ends with a model definition to save and
`llmgw ctl model add -f`, with `--ctx-size` set to the `num_ctx` Ollama ran it
with.
Ollama's prompt template is not carried: llama.cpp renders the chat template
inside the GGUF, which for any recent model is the one it was trained on. Its
system prompt and adapters are not carried either, and the import says so.
Models Ollama keeps as separate tensors rather than one GGUF, or as several
weight layers, are refused with a sentence saying so — mirror that model's GGUF
from its publisher instead. When the mirror runs beside the control plane, the
console's Mirror view does the same with **I**.
## 4. Preparing a node
**Do this before onboarding.** A node missing its driver or engine registers
@@ -458,6 +498,7 @@ Nothing to do: the agent installs them once the node joins, under
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
@@ -492,8 +533,8 @@ engines:
A changed version reaches every node when it next connects. To use an engine
you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
then installs none of its own beside it.
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
`engines.trtllm.python`): the agent then installs none of its own beside it.
#### Choosing a node's vLLM build
@@ -544,6 +585,82 @@ which drift between releases:
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
```
#### TensorRT-LLM, which you bring
llmgw does not install TensorRT-LLM. It is a pip package tied to one CUDA, one
PyTorch and an MPI the machine has to provide, and NVIDIA's own advice is its
container; an installer guessing at all of that would fail on most machines it
met, a long way into a large download. Install it on the node yourself — a
virtual environment, or NVIDIA's TensorRT-LLM container — and name its
interpreter in that node's `agent.yaml`:
```yaml
engines:
trtllm:
python: /opt/trtllm/venv/bin/python # python3 inside NVIDIA's container
```
The node then offers the engine `trtllm` whenever that interpreter has the
`tensorrt_llm` package; without the line it offers nothing, whatever is on
`PATH`. It needs **TensorRT-LLM 1.3** (1.3.0rc13 or later): 1.2 has no
`--served_model_name`, and publishes none of the queue figures the router
reads. Check a node's against the flags llmgw builds:
```
llmgw ctl engine-check trtllm --bin /opt/trtllm/venv/bin/python
```
A model for it names a Hugging Face checkpoint, as for vLLM — trtllm-serve
downloads a repository itself, or loads one from the mirror:
```yaml
name: llama-3.1-70b-instruct
source: meta-llama/Llama-3.1-70B-Instruct
profiles:
- name: h100x4
engine: trtllm
gpu_class: h100-80g
tp: 4
args: ["--max_batch_size", "64", "--free_gpu_memory_fraction", "0.85"]
```
The agent starts `trtllm-serve serve <model> --host … --port … --tp_size …
--served_model_name …`, through a few lines of Python of its own that do what
trtllm-serve has no flag for:
- **the key.** trtllm-serve checks none; every path but `/health` is made to
demand the one the gateway gives the replica, so the port on the node is not
a way around the gateway.
- **the metrics.** Its options file turns on `return_perf_metrics` and
`enable_iter_perf_stats`, without which `/prometheus/metrics` does not exist
and its queue figures do not move. The router reads
`trtllm_num_requests_running`, `trtllm_num_requests_waiting` and
`trtllm_kv_cache_utilization` there — not `/metrics`, which on trtllm-serve
is JSON. They are set as the engine iterates, so an engine gone idle shows
its last batch until the next request.
- **no usage reports to NVIDIA**: `TRTLLM_NO_USAGE_STATS` and `DO_NOT_TRACK`
are set, after the profile's own `env`.
What it does not do:
- **span nodes.** A TensorRT-LLM replica larger than one machine is launched
under MPI or Slurm from one of them, which the agents, each starting its own
node's part, do not do. A profile with `nodes` above 1 is refused; use vLLM
or SGLang for one.
- **run before Ampere.** Compute capability 8.0 or newer, which is what current
TensorRT-LLM builds kernels for.
- **take an options file from a profile.** `--config` and
`--extra_llm_api_options` are llmgw's; change a setting with `--set
PATH=VALUE` instead (`--set kv_cache_config.enable_block_reuse=false`). Also
refused, like the other engines' equivalents: `--trust_remote_code`,
`--tokenizer`, `--middleware`, `--chat_template`, `--grpc` and
`--num_serve_frontends` (both would serve without the key), and the flags the
agent sets.
This engine was written against TensorRT-LLM's documented command line,
server and metrics, and tested with stand-ins for them; it has not yet run on a
datacenter GPU. Run `engine-check` on the first node before placing a model on it.
### Weights
Nothing to do here if you built the mirror in step 3 — the agent fetches what a