Merge C23 TensorRT-LLM engine and Ollama/GGUF import

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:02:50 +02:00
commit c64af74085
28 files changed
+2376 -18

No files matched your search

+26
View File
@@ -268,6 +268,7 @@ rather than repeated.
| `models` | `[]catalog.Model` | [Server](#server) |
| `endpoints` | `[]Endpoint` | [Server](#server) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
| `store` | `string` | [Threads](#threads) |
| `retain_hours` | `int` | [Threads](#threads) |
| `dial` | `time.Duration` | [Timeouts](#timeouts) |
@@ -335,6 +336,7 @@ rather than repeated.
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
@@ -2216,6 +2218,21 @@ against sim-engines possible without a control plane.
Bin is the llmgw binary itself; the engine is one of its subcommands.
Empty means whatever started this agent, which is right almost always.
## TRTLLMConfig
### `python`
`string`
Python is the interpreter of the environment TensorRT-LLM is installed
in: <venv>/bin/python for a virtual environment, python3 in NVIDIA's
container. Naming it is what makes this node offer the engine; empty
offers nothing.
The interpreter rather than trtllm-serve, because llmgw starts the
server through a few lines of its own that make it demand the
gateway's key, which trtllm-serve has no option for.
## Threads
### `store`
@@ -2674,6 +2691,15 @@ Takes a [SGLangConfig](#sglangconfig) block.
Takes a [LlamaCppConfig](#llamacppconfig) block.
### `trtllm`
`TRTLLMConfig`
TRTLLM is TensorRT-LLM, which the agent does not install: a node
offers it only when its interpreter is named here.
Takes a [TRTLLMConfig](#trtllmconfig) block.
### `sim`
`SimConfig`