Single files were shared between the shards and replicas asking for them
at the same time; sets were not, and four replicas on one node's four cards
wrote the same .part files over one another until all but one failed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A standby follows the leader's Ready replicas over the agent port, every
two seconds, and reports back the replicas its router used and the
on-demand models it was asked for. No engine keys travel: every control
plane derives them from the cluster secret. Budgets are raised to the
store's every 30 s. Unheard from the leader for a minute, a standby
passes /v1 on as before. Off by default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
max_output on POST /api/v1/models/{name}/context, --max-output on ctl model
context, and a field in the console's Context length… form. Alone it changes
only the catalog's limit, which the router holds max_tokens to, and starts no
replica again; with a context it is set with it, never past it.
Strata's context is now changed like the others', through its setup's
--context, which writes the engine's start script at each start. Neither
needs a model removed and added again any more.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.
ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.
Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.
FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The busy figure is nvidia-smi's utilization.gpu: the share of time any
kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy
while the SMs had work 69% of the time and issued on 18% of their cycles.
Busy is now said to be that, and beside it: each card's draw against its
power limit (efficiency table, fleet and node charts, the console's GPU
work chart), and where the profiling counters can be read, SMs active, SM
occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's
GPM on Hopper and later or a dcgm-exporter on the node
(discovery.dcgm_exporter). The GPU map colours by work by default and keeps
busy as a mode. A MIG instance is never published with its card's profile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context
and the console's Context length…: the catalog, the engine's arguments and
the KV memory change together, refused past the trained context or what the
cards hold, then the replicas restart one by one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found running it through the cluster:
- Strata's server starts its engine in a process group of its own, so the
agent saw a GPU replica on the CPU and its 22 GB as somebody else's
memory. A process the shard started, at any remove, is now the shard's.
- An adopted shard's kept compute was sent at once and never again, so a
wrong one read again on adoption never reached the control plane: what
is sent is sent again when it changes, and a GPU shard kept as the CPU
is read again on adoption.
- /metrics/prometheus is llama.cpp's names without the KV cache; vLLM's,
with it, are /metrics?format=prometheus. The router now sees running 1,
waiting 2 under load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strata (github.com/Niko1221/Strata) keeps a mixture-of-experts model's
experts in RAM and the ones used most in VRAM; on cuda01's RTX 6000 it
serves Qwen3.8-Flash-Next 125B at 70-87 tokens a second to one client.
- engines/strata.go: the adapter. The agent starts llmgw strata-serve,
which runs Strata's setup the way its desktop app does
(STRATA_CONTROL_INSTALL: installs nothing, writes only under the
replica's and the node's directories) on the fetched GGUF files, points
the configuration it writes at the installed engine, and execs Strata's
server. The key goes in STRATA_API_KEY, never on a command line.
- A profile's args go to setup: --family, --size (setup's --model, which
is the source everywhere else), --context, --parallel, ... Flags that
move files or pick the engine or GPU are refused.
- provision: the source pinned by digest in a Python environment, and the
release's prebuilt engine unpacked beside it, checked against the digest
GitHub publishes: CUDA 13 (7.5+, driver 580) or CUDA 12.9 (6.0-8.9,
driver 575). Python builds can now carry release assets.
- Metrics are vLLM's names under /metrics/prometheus.
- The catalog wants one card and a family, and a directory, not one .gguf.
Opt-in: not in DefaultEngines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mirror add-peer, run on the mirror's machine, sets up a mirror on
another over ssh, kept in step with this one: llmgw there if it has
none, mirror init --peer, this mirror told of it and allowed to be read
by it, and mirror.peers in the cluster's settings. The setup screens
offer one on either database machine, as server init --second-mirror.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mirror.peers names the mirrors kept in step with mirror.api. The
mirror's management calls go to the first that can be reached, and
each mirror is asked every fifteen seconds whether it answers: a shard
is sent its model's and adapters' address on one that does while the
mirror the catalog names is down.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each mirror reads its peers every thirty seconds and copies the
versions a peer holds whole that it lacks, keeping a copy only if its
checksum is the peer's. A removal is kept with its time (.removed.json)
and listed for peers: a version removed while a peer was down is removed
there when it is back, not copied back from it; one added again after
its removal stands on both. Neither mirror leads.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
From the second review.
- An output service reads the answer with the values restore put back
taken out again (guard.Subject.Restored), as an input service reads the
request redacted.
- With n above 1, the service reads the first choice once it is whole,
not at whichever choice finished first.
- Restored values go into an answer asked for as JSON (response_format)
escaped for it, as into tool-call arguments.
- The embedding failure is logged once a minute through an outage, not
once.
- A card's own lane count and fastest generation come from the kernel
(max_link_width, max_link_speed): the driver's width.max is what card
and slot allow together, so an x16 card in an x8 slot read x8 of x8 and
was never said.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Revoking a server's tools takes them from every level beneath too, and a
level left naming no tool stays closed ([]) while any level above or
beneath names tools: an organization's revoke left its teams the copies
their first grants took, and going back to no list opened the level to
what was above it. A closed level is shown on the page and in the
console with an action to open it (store.OpenTools). A key's owner's
list is among the levels above a key.
- The flag check matches short flags as written again (-m, -hf, -mu were
let through), takes a long flag's beginning for it only for the engines
that parse with argparse (vLLM, SGLang, ComfyUI: --li is --listen), so
llama.cpp's own --chat-template is not refused, and refuses llama.cpp's
short spellings (-hfr, -hff, -mm, -jf, …).
- A stored model with an adapter by Hugging Face id can be changed again:
what it already had is not asked about anew.
- The catalog reads a Hugging Face id as the agent does (engines.IsHubID).
- The page's form cannot move llmgw:admin to another address.
- server init skips the memory check only where the gateway already runs,
not wherever a half-finished first run left server.yaml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
