TensorRT-LLM as an engine, and models from Ollama into the mirror

Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.

mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:02:01 +02:00
1 parent 30f7f3474a
commit 7940e8593c
28 files changed
+2376 -18

No files matched your search

+4
View File
@@ -329,6 +329,9 @@ var Reference = []Block{
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
}},
{Package: "engines", Name: "TRTLLMConfig", Title: "TRTLLMConfig", Doc: "", Fields: []Field{
{Key: "python", Type: "string", Doc: "Python is the interpreter of the environment TensorRT-LLM is installed\nin: <venv>/bin/python for a virtual environment, python3 in NVIDIA's\ncontainer. Naming it is what makes this node offer the engine; empty\noffers nothing.\n\nThe interpreter rather than trtllm-serve, because llmgw starts the\nserver through a few lines of its own that make it demand the\ngateway's key, which trtllm-serve has no option for.", Nested: ""},
}},
{Package: "config", Name: "Threads", Title: "Threads", Doc: "", Fields: []Field{
{Key: "store", Type: "string", Doc: "Store is the SQLite file. Empty means nothing is kept.", Nested: ""},
{Key: "retain_hours", Type: "int", Doc: "RetainHours is how long idle conversations and notes are kept. Required\nwith Store: a store with no expiry keeps everything anyone typed, by\nomission.", Nested: ""},
@@ -423,6 +426,7 @@ var Reference = []Block{
{Key: "vllm", Type: "VLLMConfig", Doc: "", Nested: "engines.VLLMConfig"},
{Key: "sglang", Type: "SGLangConfig", Doc: "", Nested: "engines.SGLangConfig"},
{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
{Key: "trtllm", Type: "TRTLLMConfig", Doc: "TRTLLM is TensorRT-LLM, which the agent does not install: a node\noffers it only when its interpreter is named here.", Nested: "engines.TRTLLMConfig"},
{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
}},
{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
+1
View File
@@ -20,6 +20,7 @@ var Sources = []string{
"internal/agent/engines/llamacpp.go",
"internal/agent/engines/vllm.go",
"internal/agent/engines/sglang.go",
"internal/agent/engines/trtllm.go",
"internal/agent/engines/sim.go",
"internal/agent/supervisor/supervisor.go",
"internal/tlsconf/tlsconf.go",