TensorRT-LLM as an engine, and models from Ollama into the mirror
Engine trtllm runs trtllm-serve on one node, through a launcher of llmgw's that demands the replica's key and turns on the Prometheus endpoint its queue and cache gauges are read from; the router reads it at /prometheus/metrics. Brought by the operator (engines.trtllm.python), not installed; checked by ctl engine-check trtllm. Written against the documented CLI, server and metrics and tested with stand-ins: not yet run on a datacenter GPU. mirror import-ollama copies a model's GGUF, and a vision model's projector, from Ollama's directory into the mirror, checked against the digest Ollama named each blob by; mirror import does the same for any local file. The console's Mirror view imports from Ollama with I. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
7940e8593c
28 files changed
+2376
-18
No files matched your search
+119
-2
@@ -383,6 +383,46 @@ llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
|
||||
or from the admin page, or by copying files into the directory by hand — the
|
||||
mirror is a directory, and anything that puts files in it works.
|
||||
|
||||
Files already on the mirror machine go in with `import`, which copies them the
|
||||
way a download is kept — under a temporary name until whole, the checksum
|
||||
beside them for nodes to check against, a version never replaced — and checks
|
||||
that a `.gguf` is one:
|
||||
|
||||
```
|
||||
llmgw mirror import ./qwen2.5-7b-instruct-q4_k_m.gguf --model qwen2.5-7b-instruct \
|
||||
--version 2026-10 --sha256 <the checksum its publisher gave>
|
||||
```
|
||||
|
||||
#### From Ollama
|
||||
|
||||
Models you already run in Ollama are GGUFs llama.cpp serves as they are.
|
||||
On the mirror machine, with Ollama's models readable there:
|
||||
|
||||
```
|
||||
llmgw mirror import-ollama # lists what Ollama has
|
||||
llmgw mirror import-ollama llama3.2:3b
|
||||
```
|
||||
|
||||
It finds Ollama's models directory where Ollama keeps it — `OLLAMA_MODELS`,
|
||||
`~/.ollama/models`, or `/usr/share/ollama/.ollama/models` for Ollama's Linux
|
||||
service, which needs `sudo` to read — or takes `--ollama-dir`. It copies the
|
||||
model's GGUF, and a vision model's projector beside it, each checked against
|
||||
the sha256 Ollama named its file by; a file that does not match is not
|
||||
mirrored, and nothing of the model is. The version is the tag and the start of
|
||||
the digest (`llama3.2/3b-a80c4f17acd5/llama3.2-3b.gguf`), because a tag like
|
||||
`latest` moves in Ollama and a mirror version must not; `--model` and
|
||||
`--version` choose otherwise. It ends with a model definition to save and
|
||||
`llmgw ctl model add -f`, with `--ctx-size` set to the `num_ctx` Ollama ran it
|
||||
with.
|
||||
|
||||
Ollama's prompt template is not carried: llama.cpp renders the chat template
|
||||
inside the GGUF, which for any recent model is the one it was trained on. Its
|
||||
system prompt and adapters are not carried either, and the import says so.
|
||||
Models Ollama keeps as separate tensors rather than one GGUF, or as several
|
||||
weight layers, are refused with a sentence saying so — mirror that model's GGUF
|
||||
from its publisher instead. When the mirror runs beside the control plane, the
|
||||
console's Mirror view does the same with **I**.
|
||||
|
||||
## 4. Preparing a node
|
||||
|
||||
**Do this before onboarding.** A node missing its driver or engine registers
|
||||
@@ -458,6 +498,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
|
||||
|
||||
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
|
||||
node gets the same dependencies whenever it is built, and installed with
|
||||
@@ -492,8 +533,8 @@ engines:
|
||||
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
|
||||
`engines.trtllm.python`): the agent then installs none of its own beside it.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
@@ -544,6 +585,82 @@ which drift between releases:
|
||||
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
|
||||
```
|
||||
|
||||
#### TensorRT-LLM, which you bring
|
||||
|
||||
llmgw does not install TensorRT-LLM. It is a pip package tied to one CUDA, one
|
||||
PyTorch and an MPI the machine has to provide, and NVIDIA's own advice is its
|
||||
container; an installer guessing at all of that would fail on most machines it
|
||||
met, a long way into a large download. Install it on the node yourself — a
|
||||
virtual environment, or NVIDIA's TensorRT-LLM container — and name its
|
||||
interpreter in that node's `agent.yaml`:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
trtllm:
|
||||
python: /opt/trtllm/venv/bin/python # python3 inside NVIDIA's container
|
||||
```
|
||||
|
||||
The node then offers the engine `trtllm` whenever that interpreter has the
|
||||
`tensorrt_llm` package; without the line it offers nothing, whatever is on
|
||||
`PATH`. It needs **TensorRT-LLM 1.3** (1.3.0rc13 or later): 1.2 has no
|
||||
`--served_model_name`, and publishes none of the queue figures the router
|
||||
reads. Check a node's against the flags llmgw builds:
|
||||
|
||||
```
|
||||
llmgw ctl engine-check trtllm --bin /opt/trtllm/venv/bin/python
|
||||
```
|
||||
|
||||
A model for it names a Hugging Face checkpoint, as for vLLM — trtllm-serve
|
||||
downloads a repository itself, or loads one from the mirror:
|
||||
|
||||
```yaml
|
||||
name: llama-3.1-70b-instruct
|
||||
source: meta-llama/Llama-3.1-70B-Instruct
|
||||
profiles:
|
||||
- name: h100x4
|
||||
engine: trtllm
|
||||
gpu_class: h100-80g
|
||||
tp: 4
|
||||
args: ["--max_batch_size", "64", "--free_gpu_memory_fraction", "0.85"]
|
||||
```
|
||||
|
||||
The agent starts `trtllm-serve serve <model> --host … --port … --tp_size …
|
||||
--served_model_name …`, through a few lines of Python of its own that do what
|
||||
trtllm-serve has no flag for:
|
||||
|
||||
- **the key.** trtllm-serve checks none; every path but `/health` is made to
|
||||
demand the one the gateway gives the replica, so the port on the node is not
|
||||
a way around the gateway.
|
||||
- **the metrics.** Its options file turns on `return_perf_metrics` and
|
||||
`enable_iter_perf_stats`, without which `/prometheus/metrics` does not exist
|
||||
and its queue figures do not move. The router reads
|
||||
`trtllm_num_requests_running`, `trtllm_num_requests_waiting` and
|
||||
`trtllm_kv_cache_utilization` there — not `/metrics`, which on trtllm-serve
|
||||
is JSON. They are set as the engine iterates, so an engine gone idle shows
|
||||
its last batch until the next request.
|
||||
- **no usage reports to NVIDIA**: `TRTLLM_NO_USAGE_STATS` and `DO_NOT_TRACK`
|
||||
are set, after the profile's own `env`.
|
||||
|
||||
What it does not do:
|
||||
|
||||
- **span nodes.** A TensorRT-LLM replica larger than one machine is launched
|
||||
under MPI or Slurm from one of them, which the agents, each starting its own
|
||||
node's part, do not do. A profile with `nodes` above 1 is refused; use vLLM
|
||||
or SGLang for one.
|
||||
- **run before Ampere.** Compute capability 8.0 or newer, which is what current
|
||||
TensorRT-LLM builds kernels for.
|
||||
- **take an options file from a profile.** `--config` and
|
||||
`--extra_llm_api_options` are llmgw's; change a setting with `--set
|
||||
PATH=VALUE` instead (`--set kv_cache_config.enable_block_reuse=false`). Also
|
||||
refused, like the other engines' equivalents: `--trust_remote_code`,
|
||||
`--tokenizer`, `--middleware`, `--chat_template`, `--grpc` and
|
||||
`--num_serve_frontends` (both would serve without the key), and the flags the
|
||||
agent sets.
|
||||
|
||||
This engine was written against TensorRT-LLM's documented command line,
|
||||
server and metrics, and tested with stand-ins for them; it has not yet run on a
|
||||
datacenter GPU. Run `engine-check` on the first node before placing a model on it.
|
||||
|
||||
### Weights
|
||||
|
||||
Nothing to do here if you built the mirror in step 3 — the agent fetches what a
|
||||
|
||||
Reference in new issue
Block a user