llmgw/notes/c23.md

CHANGELOG

Engines and the mirror

• TensorRT-LLM as an engine (engine: trtllm), on nodes where you installed it: name its interpreter in the node's agent.yaml (engines.trtllm.python) and the node offers it while that interpreter has the tensorrt_llm package. llmgw does not install it. The agent starts trtllm-serve serve with the replica's host, port, --tp_size, --pp_size and served name, through a few lines of its own that make it demand the gateway's key on every path but /health (trtllm-serve has no key option), turn on the Prometheus endpoint and iteration statistics, and keep usage reports to NVIDIA off. The router reads it at /prometheus/metrics — trtllm_num_requests_running, trtllm_num_requests_waiting, trtllm_kv_cache_utilization, tokens and successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer GPU; one node per replica. Profiles may not give it an options file, --trust_remote_code, --grpc, --num_serve_frontends or the flags the agent sets; --set PATH=VALUE changes a setting instead. llmgw ctl engine-check trtllm --bin <python> checks an installed one. Written against TensorRT-LLM's documented command line, server and metrics and tested with stand-ins for them; not yet run on a datacenter GPU.
• llmgw mirror import-ollama [NAME[:TAG]] puts a model you already have in Ollama into the mirror for llama.cpp: its GGUF, and a vision model's projector beside it, each checked against the sha256 Ollama named its file by. With no name it lists Ollama's models; it finds Ollama's directory itself (OLLAMA_MODELS, ~/.ollama/models, Ollama's Linux service) or takes --ollama-dir. The version is the tag and the start of the digest, since a tag moves in Ollama. It prints a model definition to add, with Ollama's num_ctx as --ctx-size. The console's Mirror view does the same with I, for the mirror beside the control plane.
• llmgw mirror import FILE... --model NAME copies files already on the mirror machine into it the way a download is kept: whole or not at all, the checksum beside it, a version never replaced, a .gguf checked to be one, and --sha256 checked when given.

BACKLOG

Done (branch features-c2): engine trtllm (internal/agent/engines/trtllm.go): trtllm-serve through a launcher of llmgw's that demands the replica's key (trtllm-serve has none) and writes the options file that turns on /prometheus/metrics and the iteration statistics; the scrape contract for it (scrape.TRTLLMMapping, with a per-engine metrics path); found, not installed — engines.trtllm.python in agent.yaml, probed from the package's metadata; catalog validation (GPU class, one node, no GGUF source, the flags a profile may not give); ctl engine-check trtllm, which asks trtllm-serve serve --help through its module. Ollama/GGUF import: mirror import-ollama (manifest parsing, blob digest check, projector beside the model, all or nothing, a model definition printed), mirror import for any local file, and the console's I on the Mirror view. Left: TensorRT-LLM has not run on real hardware — the first node with one should run engine-check and serve a model before anyone relies on it; no installer for it; no multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one node); Triton Inference Server as an engine is not done; trtllm's queue gauges are set per iteration, so an idle engine shows its last batch until the next request; --num_serve_frontends is refused because extra front ends would not demand the key. The Ollama import runs on the mirror machine only — the mirror's API takes no files, so an Ollama elsewhere means copying the blob or running the import there; Ollama models kept as separate tensors or as several weight layers, and its LoRA adapters, are refused or left with a sentence; files an import run with sudo writes are root's, as mirror add's are.