Commit Graph
4 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 7940e8593c TensorRT-LLM as an engine, and models from Ollama into the mirror
Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.

mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:02:01 +02:00
Ilker ManapandClaude Opus 5.5 f1b49d0251 Bound idle time, header size and request reading on every listener, and connections on the client port
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:21:15 +02:00
Ilker ManapandClaude Opus 5.5 c19abe95e1 Refuse profile env and args that would run code or touch files on the nodes
env is an allowlist of tuning families less path, plugin, library, address
and credential names (NCCL_*_PLUGIN, CUDA_INJECTION64_PATH and the like
got through the old denylist); llama.cpp's LLAMA_ARG_* are checked as
flags. args refuse --trust-remote-code, --log-file, --slot-save-path,
--tool-parser-plugin, --middleware, --allowed-local-media-path and others
that read, write or load, and what the agent sets itself. Checked in the
catalog and again on the node before an engine starts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 09:54:45 +02:00
Ilker ManapandClaude Opus 5.5 d167686cf1 Stop a browser page, or a model definition, from running code on the nodes
The admin listener has no login. Reaching it is meant to mean holding the
control plane, so it binds to loopback and an operator reaches it over an
ssh tunnel — and so does every page open in that operator's browser. The
API decoded JSON whatever the Content-Type said, so a page could POST a
text/plain model definition, no preflight needed. A model's profile env
reached an engine the agent runs as root, unfiltered: LD_PRELOAD in a
definition was code execution on every node. A page that rebound its own
name to 127.0.0.1 could also read pprof's heap, which holds API keys.

The admin listener now answers only when Host is an address or localhost,
refuses requests a browser marks as cross-site (Origin, Sec-Fetch-Site),
and takes changes only as application/json, which no page can send across
origins without a preflight nobody grants. ctl sends that on every change,
bodiless DELETEs included; curl and ctl are otherwise unaffected.

Profile env that changes what runs rather than how — the dynamic loader's
LD_*, PATH, Python's PYTHON* start-up variables, BASH_ENV and their like —
is refused when a model is defined, and dropped again by the engine adapter
before the process starts, whatever sent the profile. None of the live
catalog's models sets any.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 00:29:52 +02:00