TensorRT-LLM as an engine, and models from Ollama into the mirror
Engine trtllm runs trtllm-serve on one node, through a launcher of llmgw's that demands the replica's key and turns on the Prometheus endpoint its queue and cache gauges are read from; the router reads it at /prometheus/metrics. Brought by the operator (engines.trtllm.python), not installed; checked by ctl engine-check trtllm. Written against the documented CLI, server and metrics and tested with stand-ins: not yet run on a datacenter GPU. mirror import-ollama copies a model's GGUF, and a vision model's projector, from Ollama's directory into the mirror, checked against the digest Ollama named each blob by; mirror import does the same for any local file. The console's Mirror view imports from Ollama with I. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
7940e8593c
28 files changed
+2376
-18
No files matched your search
@@ -0,0 +1,61 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Engines and the mirror
|
||||
|
||||
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
|
||||
installed it: name its interpreter in the node's `agent.yaml`
|
||||
(`engines.trtllm.python`) and the node offers it while that interpreter
|
||||
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
|
||||
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
|
||||
`--pp_size` and served name, through a few lines of its own that make it
|
||||
demand the gateway's key on every path but `/health` (trtllm-serve has no
|
||||
key option), turn on the Prometheus endpoint and iteration statistics, and
|
||||
keep usage reports to NVIDIA off. The router reads it at
|
||||
`/prometheus/metrics` — `trtllm_num_requests_running`,
|
||||
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
|
||||
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
|
||||
GPU; one node per replica. Profiles may not give it an options file,
|
||||
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
|
||||
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
|
||||
engine-check trtllm --bin <python>` checks an installed one. Written
|
||||
against TensorRT-LLM's documented command line, server and metrics and
|
||||
tested with stand-ins for them; not yet run on a datacenter GPU.
|
||||
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
|
||||
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
|
||||
model's projector beside it, each checked against the sha256 Ollama named
|
||||
its file by. With no name it lists Ollama's models; it finds Ollama's
|
||||
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
|
||||
service) or takes `--ollama-dir`. The version is the tag and the start of
|
||||
the digest, since a tag moves in Ollama. It prints a model definition to
|
||||
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
|
||||
view does the same with **I**, for the mirror beside the control plane.
|
||||
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
|
||||
mirror machine into it the way a download is kept: whole or not at all, the
|
||||
checksum beside it, a version never replaced, a `.gguf` checked to be one,
|
||||
and `--sha256` checked when given.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** engine `trtllm` (`internal/agent/engines/trtllm.go`):
|
||||
`trtllm-serve` through a launcher of llmgw's that demands the replica's key
|
||||
(trtllm-serve has none) and writes the options file that turns on
|
||||
`/prometheus/metrics` and the iteration statistics; the scrape contract for
|
||||
it (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
|
||||
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
|
||||
package's metadata; catalog validation (GPU class, one node, no GGUF source,
|
||||
the flags a profile may not give); `ctl engine-check trtllm`, which asks
|
||||
`trtllm-serve serve --help` through its module. Ollama/GGUF import: `mirror
|
||||
import-ollama` (manifest parsing, blob digest check, projector beside the
|
||||
model, all or nothing, a model definition printed), `mirror import` for any
|
||||
local file, and the console's **I** on the Mirror view. Left: TensorRT-LLM has
|
||||
not run on real hardware — the first node with one should run `engine-check`
|
||||
and serve a model before anyone relies on it; no installer for it; no
|
||||
multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one
|
||||
node); Triton Inference Server as an engine is not done; trtllm's queue gauges
|
||||
are set per iteration, so an idle engine shows its last batch until the next
|
||||
request; `--num_serve_frontends` is refused because extra front ends would not
|
||||
demand the key. The Ollama import runs on the mirror machine only — the
|
||||
mirror's API takes no files, so an Ollama elsewhere means copying the blob or
|
||||
running the import there; Ollama models kept as separate tensors or as several
|
||||
weight layers, and its LoRA adapters, are refused or left with a sentence;
|
||||
files an import run with `sudo` writes are root's, as `mirror add`'s are.
|
||||
Reference in new issue
Block a user