Found running it through the cluster:
- Strata's server starts its engine in a process group of its own, so the
agent saw a GPU replica on the CPU and its 22 GB as somebody else's
memory. A process the shard started, at any remove, is now the shard's.
- An adopted shard's kept compute was sent at once and never again, so a
wrong one read again on adoption never reached the control plane: what
is sent is sent again when it changes, and a GPU shard kept as the CPU
is read again on adoption.
- /metrics/prometheus is llama.cpp's names without the KV cache; vLLM's,
with it, are /metrics?format=prometheus. The router now sees running 1,
waiting 2 under load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strata (github.com/Niko1221/Strata) keeps a mixture-of-experts model's
experts in RAM and the ones used most in VRAM; on cuda01's RTX 6000 it
serves Qwen3.8-Flash-Next 125B at 70-87 tokens a second to one client.
- engines/strata.go: the adapter. The agent starts llmgw strata-serve,
which runs Strata's setup the way its desktop app does
(STRATA_CONTROL_INSTALL: installs nothing, writes only under the
replica's and the node's directories) on the fetched GGUF files, points
the configuration it writes at the installed engine, and execs Strata's
server. The key goes in STRATA_API_KEY, never on a command line.
- A profile's args go to setup: --family, --size (setup's --model, which
is the source everywhere else), --context, --parallel, ... Flags that
move files or pick the engine or GPU are refused.
- provision: the source pinned by digest in a Python environment, and the
release's prebuilt engine unpacked beside it, checked against the digest
GitHub publishes: CUDA 13 (7.5+, driver 580) or CUDA 12.9 (6.0-8.9,
driver 575). Python builds can now carry release assets.
- Metrics are vLLM's names under /metrics/prometheus.
- The catalog wants one card and a family, and a directory, not one .gguf.
Opt-in: not in DefaultEngines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>