Merge C23 TensorRT-LLM engine and Ollama/GGUF import
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
c64af74085
28 files changed
+2376
-18
No files matched your search
@@ -268,6 +268,7 @@ rather than repeated.
|
||||
| `models` | `[]catalog.Model` | [Server](#server) |
|
||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
| `store` | `string` | [Threads](#threads) |
|
||||
| `retain_hours` | `int` | [Threads](#threads) |
|
||||
| `dial` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
@@ -335,6 +336,7 @@ rather than repeated.
|
||||
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
||||
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -2216,6 +2218,21 @@ against sim-engines possible without a control plane.
|
||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||
Empty means whatever started this agent, which is right almost always.
|
||||
|
||||
## TRTLLMConfig
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of the environment TensorRT-LLM is installed
|
||||
in: <venv>/bin/python for a virtual environment, python3 in NVIDIA's
|
||||
container. Naming it is what makes this node offer the engine; empty
|
||||
offers nothing.
|
||||
|
||||
The interpreter rather than trtllm-serve, because llmgw starts the
|
||||
server through a few lines of its own that make it demand the
|
||||
gateway's key, which trtllm-serve has no option for.
|
||||
|
||||
## Threads
|
||||
|
||||
### `store`
|
||||
@@ -2674,6 +2691,15 @@ Takes a [SGLangConfig](#sglangconfig) block.
|
||||
|
||||
Takes a [LlamaCppConfig](#llamacppconfig) block.
|
||||
|
||||
### `trtllm`
|
||||
|
||||
`TRTLLMConfig`
|
||||
|
||||
TRTLLM is TensorRT-LLM, which the agent does not install: a node
|
||||
offers it only when its interpreter is named here.
|
||||
|
||||
Takes a [TRTLLMConfig](#trtllmconfig) block.
|
||||
|
||||
### `sim`
|
||||
|
||||
`SimConfig`
|
||||
|
||||
Reference in new issue
Block a user