Files
llmgw/notes/c23.md
T
Ilker ManapandClaude Opus 5.5 7940e8593c TensorRT-LLM as an engine, and models from Ollama into the mirror
Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.

mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:02:01 +02:00

62 lines
3.9 KiB
Markdown

## CHANGELOG
### Engines and the mirror
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
installed it: name its interpreter in the node's `agent.yaml`
(`engines.trtllm.python`) and the node offers it while that interpreter
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
`--pp_size` and served name, through a few lines of its own that make it
demand the gateway's key on every path but `/health` (trtllm-serve has no
key option), turn on the Prometheus endpoint and iteration statistics, and
keep usage reports to NVIDIA off. The router reads it at
`/prometheus/metrics` — `trtllm_num_requests_running`,
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
GPU; one node per replica. Profiles may not give it an options file,
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
engine-check trtllm --bin <python>` checks an installed one. Written
against TensorRT-LLM's documented command line, server and metrics and
tested with stand-ins for them; not yet run on a datacenter GPU.
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
model's projector beside it, each checked against the sha256 Ollama named
its file by. With no name it lists Ollama's models; it finds Ollama's
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
service) or takes `--ollama-dir`. The version is the tag and the start of
the digest, since a tag moves in Ollama. It prints a model definition to
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
view does the same with **I**, for the mirror beside the control plane.
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
mirror machine into it the way a download is kept: whole or not at all, the
checksum beside it, a version never replaced, a `.gguf` checked to be one,
and `--sha256` checked when given.
## BACKLOG
**Done (branch `features-c2`):** engine `trtllm` (`internal/agent/engines/trtllm.go`):
`trtllm-serve` through a launcher of llmgw's that demands the replica's key
(trtllm-serve has none) and writes the options file that turns on
`/prometheus/metrics` and the iteration statistics; the scrape contract for
it (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
package's metadata; catalog validation (GPU class, one node, no GGUF source,
the flags a profile may not give); `ctl engine-check trtllm`, which asks
`trtllm-serve serve --help` through its module. Ollama/GGUF import: `mirror
import-ollama` (manifest parsing, blob digest check, projector beside the
model, all or nothing, a model definition printed), `mirror import` for any
local file, and the console's **I** on the Mirror view. Left: TensorRT-LLM has
not run on real hardware — the first node with one should run `engine-check`
and serve a model before anyone relies on it; no installer for it; no
multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one
node); Triton Inference Server as an engine is not done; trtllm's queue gauges
are set per iteration, so an idle engine shows its last batch until the next
request; `--num_serve_frontends` is refused because extra front ends would not
demand the key. The Ollama import runs on the mirror machine only — the
mirror's API takes no files, so an Ollama elsewhere means copying the blob or
running the import there; Ollama models kept as separate tensors or as several
weight layers, and its LoRA adapters, are refused or left with a sentence;
files an import run with `sudo` writes are root's, as `mirror add`'s are.