Engine trtllm runs trtllm-serve on one node, through a launcher of llmgw's that demands the replica's key and turns on the Prometheus endpoint its queue and cache gauges are read from; the router reads it at /prometheus/metrics. Brought by the operator (engines.trtllm.python), not installed; checked by ctl engine-check trtllm. Written against the documented CLI, server and metrics and tested with stand-ins: not yet run on a datacenter GPU. mirror import-ollama copies a model's GGUF, and a vision model's projector, from Ollama's directory into the mirror, checked against the digest Ollama named each blob by; mirror import does the same for any local file. The console's Mirror view imports from Ollama with I. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
62 lines
3.9 KiB
Markdown
62 lines
3.9 KiB
Markdown
## CHANGELOG
|
|
|
|
### Engines and the mirror
|
|
|
|
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
|
|
installed it: name its interpreter in the node's `agent.yaml`
|
|
(`engines.trtllm.python`) and the node offers it while that interpreter
|
|
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
|
|
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
|
|
`--pp_size` and served name, through a few lines of its own that make it
|
|
demand the gateway's key on every path but `/health` (trtllm-serve has no
|
|
key option), turn on the Prometheus endpoint and iteration statistics, and
|
|
keep usage reports to NVIDIA off. The router reads it at
|
|
`/prometheus/metrics` — `trtllm_num_requests_running`,
|
|
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
|
|
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
|
|
GPU; one node per replica. Profiles may not give it an options file,
|
|
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
|
|
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
|
|
engine-check trtllm --bin <python>` checks an installed one. Written
|
|
against TensorRT-LLM's documented command line, server and metrics and
|
|
tested with stand-ins for them; not yet run on a datacenter GPU.
|
|
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
|
|
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
|
|
model's projector beside it, each checked against the sha256 Ollama named
|
|
its file by. With no name it lists Ollama's models; it finds Ollama's
|
|
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
|
|
service) or takes `--ollama-dir`. The version is the tag and the start of
|
|
the digest, since a tag moves in Ollama. It prints a model definition to
|
|
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
|
|
view does the same with **I**, for the mirror beside the control plane.
|
|
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
|
|
mirror machine into it the way a download is kept: whole or not at all, the
|
|
checksum beside it, a version never replaced, a `.gguf` checked to be one,
|
|
and `--sha256` checked when given.
|
|
|
|
## BACKLOG
|
|
|
|
**Done (branch `features-c2`):** engine `trtllm` (`internal/agent/engines/trtllm.go`):
|
|
`trtllm-serve` through a launcher of llmgw's that demands the replica's key
|
|
(trtllm-serve has none) and writes the options file that turns on
|
|
`/prometheus/metrics` and the iteration statistics; the scrape contract for
|
|
it (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
|
|
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
|
|
package's metadata; catalog validation (GPU class, one node, no GGUF source,
|
|
the flags a profile may not give); `ctl engine-check trtllm`, which asks
|
|
`trtllm-serve serve --help` through its module. Ollama/GGUF import: `mirror
|
|
import-ollama` (manifest parsing, blob digest check, projector beside the
|
|
model, all or nothing, a model definition printed), `mirror import` for any
|
|
local file, and the console's **I** on the Mirror view. Left: TensorRT-LLM has
|
|
not run on real hardware — the first node with one should run `engine-check`
|
|
and serve a model before anyone relies on it; no installer for it; no
|
|
multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one
|
|
node); Triton Inference Server as an engine is not done; trtllm's queue gauges
|
|
are set per iteration, so an idle engine shows its last batch until the next
|
|
request; `--num_serve_frontends` is refused because extra front ends would not
|
|
demand the key. The Ollama import runs on the mirror machine only — the
|
|
mirror's API takes no files, so an Ollama elsewhere means copying the blob or
|
|
running the import there; Ollama models kept as separate tensors or as several
|
|
weight layers, and its LoRA adapters, are refused or left with a sentence;
|
|
files an import run with `sudo` writes are root's, as `mirror add`'s are.
|