TensorRT-LLM as an engine, and models from Ollama into the mirror
Engine trtllm runs trtllm-serve on one node, through a launcher of llmgw's that demands the replica's key and turns on the Prometheus endpoint its queue and cache gauges are read from; the router reads it at /prometheus/metrics. Brought by the operator (engines.trtllm.python), not installed; checked by ctl engine-check trtllm. Written against the documented CLI, server and metrics and tested with stand-ins: not yet run on a datacenter GPU. mirror import-ollama copies a model's GGUF, and a vision model's projector, from Ollama's directory into the mirror, checked against the digest Ollama named each blob by; mirror import does the same for any local file. The console's Mirror view imports from Ollama with I. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
7940e8593c
28 files changed
+2376
-18
No files matched your search
@@ -264,6 +264,7 @@ rather than repeated.
|
||||
| `models` | `[]catalog.Model` | [Server](#server) |
|
||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
| `store` | `string` | [Threads](#threads) |
|
||||
| `retain_hours` | `int` | [Threads](#threads) |
|
||||
| `dial` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
@@ -331,6 +332,7 @@ rather than repeated.
|
||||
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
||||
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -2175,6 +2177,21 @@ against sim-engines possible without a control plane.
|
||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||
Empty means whatever started this agent, which is right almost always.
|
||||
|
||||
## TRTLLMConfig
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of the environment TensorRT-LLM is installed
|
||||
in: <venv>/bin/python for a virtual environment, python3 in NVIDIA's
|
||||
container. Naming it is what makes this node offer the engine; empty
|
||||
offers nothing.
|
||||
|
||||
The interpreter rather than trtllm-serve, because llmgw starts the
|
||||
server through a few lines of its own that make it demand the
|
||||
gateway's key, which trtllm-serve has no option for.
|
||||
|
||||
## Threads
|
||||
|
||||
### `store`
|
||||
@@ -2633,6 +2650,15 @@ Takes a [SGLangConfig](#sglangconfig) block.
|
||||
|
||||
Takes a [LlamaCppConfig](#llamacppconfig) block.
|
||||
|
||||
### `trtllm`
|
||||
|
||||
`TRTLLMConfig`
|
||||
|
||||
TRTLLM is TensorRT-LLM, which the agent does not install: a node
|
||||
offers it only when its interpreter is named here.
|
||||
|
||||
Takes a [TRTLLMConfig](#trtllmconfig) block.
|
||||
|
||||
### `sim`
|
||||
|
||||
`SimConfig`
|
||||
|
||||
Reference in new issue
Block a user