An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CI run 394, on a commit that changed only BACKLOG.md, failed with a data
race in TestAVisionModelIsServedWithItsProjectorBesideIt. The serve job's
goroutine wrote j.Request without the lock (since efffdb6, which sized
embedding slots), while Get's snapshot read the job under it. Every other
write to a running job holds the lock. It did not reproduce in 30 runs
here; the race detector's trace names this write and that read.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A v0.14.6 install on fewer than eight cores refused every request until
its judge was given cores by hand, and upgrading did not change that. The
judge keeper, on the leader, now finds a judge whose entry is exactly what
init before v0.14.7 wrote: one llama.cpp CPU profile, an 8192 context and
nothing else, 2 or 4 GB, none to two cores. It gives the judge 2 GB,
--cache-ram 0 and cores by init's rule through modelsvc.SetProfiles, then
restarts its ready replicas. The replica count is left as it is. A judge
whose entry differs is left alone, and the log says so once.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a 4-core single-machine install, `ctl model resources` was run on the
judge (to two cores) and on the starter (to none), then both restarted.
The judge's restart was refused: its replacement fit on no node even with
its old replica stopped, because the starter still held its two cores. A
restart now waits for other restarts under way (queued, retiring or
draining) when their going would make room, up to DrainGrace.
A drained replica was also kept until its model had enough live replicas,
and a replica being restarted that still answers did not count. So with
replicas restarted one after another, each drained one waited out the
five-minute grace, holding its room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Measured again after the store fixes. Cutting the store primary's power no
longer hung requests, but leadership moved anyway: the lease had lapsed for
everyone while Patroni promoted the standby, and a standby asked first. The
new leader then spent about 25 s without its nodes, and so without its
judges.
A standby that saw the store out of reach now leaves the lease alone for one
renew round after the store returns. The default lease is 10 s, and a node
detects a control plane that died silently in 8 s (5 s idle, 3 s ping)
rather than 15.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
140 named 150 a store member, 150 named itself a control plane: Prometheus
asking both scraped 150 twice and counted its every series twice, and asked
of 140 alone while 150 led it lost every node. Each control plane now lists
all of them as control planes before the store's members, so two answers
agree and a target is scraped once. The shipped prometheus.yml says to
scrape and ask every control plane.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Apertus 70B with a 65536-token pool, and then 8B with 131072, on a 24 GB
card loaded their weights and failed to make their context on every start,
the 8B taking the card from the 70B each time only to fail. ctl model serve
now shrinks a pool the card cannot hold beside the weights (KV cache from
the GGUF's shape, 90% of the card, a gigabyte for the engine), saying so,
and refuses weights that do not fit at all; the portal's serve form refuses
such a pool and says the largest that fits.
The mirror page removed the 70B's weights while the catalog still had the
model, whose replicas went on loading the node's copy with nothing said.
Removing weights a catalog model loads is refused, naming the model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
flux and sf3d on one card handed it to each other, each starting ComfyUI
and loading its weights from nothing (BACKLOG H3, placement side). An
environment of two or more models now gets a hidden catalog entry,
comfyui-env-<name>: on its first member's profiles with the most memory
any member needs, on demand for the longest idle timeout, carrying every
member's weights, repositories and the environment's custom nodes. The
members get no replicas of their own; asking for one wants the entry.
The agent fetches each member's weights; the gate links all of them,
refusing two weights files of one name, links each member's Hugging Face
repository, and answers /v1/models for every member. The control plane
offers the replica under the members' names with each one's workflow and
checkpoint, which the router sends by the model asked for. A member's
sibling never evicts the replica serving it, and the members' load counts
as the entry's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl model serve served one as a chat model: no --embedding, four slots, a
512-token batch, embeddings unset, and a chat smoke test it could not
pass. A GGUF that pools (<arch>.pooling_type), or --embeddings, gets
--embedding, 32 slots of 1024 tokens unless sized otherwise, a batch that
holds a whole passage, the embeddings capability, and one vector asked for
as its smoke test (canaries and updates too). An import of a hundred
thousand chunks had gone at the pace of four slots with the card a third
busy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On one card shared by flux and sf3d, a mesh asked for after a picture
waited out the picture model's five minutes of protection and two of idle.
on_demand.evict_after_idle and on_demand.min_lifetime set those for the
cluster, in force on the leader as they change; unset keeps 2m and 5m.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A customer's portal had no Images or 3D pages: nobody had written a
definition for either, on a cluster whose V100 runs both. The leading
control plane now adds flux-schnell and triposr, on demand, once a node
has a card that fits one, the cluster installs ComfyUI, and the catalog
makes nothing of that kind. presets.auto (a live setting, on by default)
turns it off; a preset somebody removes is recorded as declined and not
added back until added again by hand. Each addition is in the audit log.
GET /api/v1/presets says how each preset stands, the ComfyUI the cluster
installs, and why one is not added; POST /api/v1/presets/{name} adds one
(setting engines.comfyui where there is none); PUT /api/v1/presets/auto
switches it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
flux-schnell and triposr come with llmgw as presets: their workflow,
custom nodes, packages and licence as they were found and tried on the
PoC, and a profile that says the card it needs rather than a class, so the
entry places on any site's cards. TripoSR's workflow is built in, with the
LLMGWPrepareForMesh node that keeps a picture's background out of the
mesh, which the copy in the document had left out.
`ctl model add PRESET` adds one, and has the nodes install ComfyUI when
the cluster's settings name none; `ctl model presets` lists them; the
console's add-model dialog offers them beside a definition file. A
ComfyUI build is now also checked to keep its order on a Turing card, so
a node's installed cu13 is not replaced by the cu126 build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VLM is to come first on the one card. Its class makes it take the card
from flux, triposr or sf3d when asked for, but idle eviction ignored class:
a 3D job two minutes into a lull would have taken the card from it. Idle
eviction offers only replicas of no higher class than the model wanting
room, and the wait such a model is told runs to the holder's idle timeout.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Removed and added again to give it a licence, qwen3-vl-8b came back loading
by itself: its name was still on the wanted list from before. A model that
leaves the catalog leaves the list too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stable Fast 3D's texture_baker and uv_unwrapper are C++/CUDA packages no
index has; on the PoC they were built and installed into ComfyUI's
environment by hand. nodes[].build names such directories, and the gate
builds each as a wheel with ComfyUI's Python, for the compute capability
torch reports for the replica's own card, keeps it on the node by the
node's commit, the architecture and the directory, and installs it
beside the replica with the node's requirements. A copy in ComfyUI's
environment does not pin it. A node with no CUDA toolkit says so.
comfyui.huggingface_repo shows the replica's weights to a node that
loads them by repository name, in the engines' Hugging Face cache, so a
gated repository needs no token on the node.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A catalog entry meant for many sites cannot know their GPU class names:
the PoC has a Quadro RTX 6000, a customer an A100, another an L40S. A
profile may now say gpu: {min_compute_capability, min_memory_gb} in
place of gpu_class, and stands for every class in the cluster that meets
it, named <profile>@<class>, at its own priority; placement still sees
exact classes. ProfileByName and EngineProfile resolve such a name to
the profile on that class, so every caller that maps a replica back to
its profile works unchanged. A ComfyUI profile may carry its own
workflow, checkpoint and nodes, and with its own source an fp8 workflow
and weights go to cards with fp8 and fp16 ones to Turing.
A profile by need is left out of placement until the cluster's cards are
known: with no class it would have read as a CPU profile.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's catalog entry now says what its workflow takes and makes:
comfyui.inputs, each a placeholder of its name with a type (text, int,
float, bool, choice, image, mask, audio, video, file), bounds, choices
and a default; and comfyui.output, which gains video, audio and files
beside pictures and meshes. POST /v1/workflows/runs with {model, inputs}
checks a request against them before a job is written — a required input
left out, an unknown one, a number out of bounds, a song sent as a
picture, another owner's file id are each refused with what was wrong —
and the job worker replays it to a replica, where the files go into
ComfyUI's input directory, the workflow is filled and run, and the first
file of the model's output kind is the job's result. /v1/models lists
each ComfyUI model's workflow, so a client can build its form.
A PNG result carries the licence in an iTXt chunk, as a GLB does in
asset.extras. A file whose server named no type is typed by its name
before its bytes, so a WAV is audio/wav, not audio/wave.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A ComfyUI model may name an environment: the models of one environment
agree on every custom node they both name and on every package range,
so one replica can carry all of them and ComfyUI moves between their
weights itself rather than the card being handed over with a cold start
each time. The catalog refuses an environment that cannot share a
Python, naming both models; Catalog.Environments gives each one's
models, the union of their nodes and their weight sources, for
placement to build on.
A licence gains an id (SPDX where there is one), whether its output may
be used commercially, the regions it excludes, a revenue cap and the
attribution it asks for, so a person's or a site's choice of licences
can be matched without reading prose.
BACKLOG gains section H: ComfyUI as a platform.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After a control plane restart flux, adopted and asked for by nobody, had no
last use and so was never idle enough to evict: a 3D job waited out its
whole fifteen minutes with no estimate. It counts as used at adoption, as it
is already counted as wanted then; a replica in use is reported by the
router within seconds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A 3D job waiting for the picture model to give up the one card had an empty
progress; its client could not tell waiting from stuck. The control plane
says which model holds the cards and when it lets go, plus the waiting
model's last start; the 503 carries it as headers, and a job keeps it as
progress {state: waiting_gpu, held_by, eta_seconds}.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
flux evicted to make room for triposr was placed again two seconds later:
still on the wanted list, and first in the catalog, it took back the card
it had just given up, and the 3D job went on waiting. Its last replica
evicted, it leaves the list until it is asked for again.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A 3D job asked for straight after a picture, on a cluster with one card,
failed at once with "cannot be placed": flux held the card, too freshly
started to evict. The cards coming back when an on-demand model goes idle
makes it a wait, so the request is told to retry and the job waits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/v1/openapi.json is the API clients use — OpenAI's endpoints, Anthropic's,
and what llmgw adds — in OpenAPI 3.1, generated from internal/clientapi's
route table by the generator the admin API's document comes from, now
told each document's title, server, token and error. A test reads the
router's mux.HandleFunc calls and refuses a route described and not
mounted, or mounted and not described. Asked for before there is a key,
so it needs none.
/docs/api shows it in Swagger UI 5.33.1 (Apache 2.0, served from the
gateway as the page's policy requires): Authorize takes a key, Try it out
sends to this gateway. Staff are offered the admin API's document too,
from /docs/api/admin.json, which nobody else is given.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/3d beside /images, while some model makes meshes: a picture chosen, the
job asked for through the same path an API client's takes, its progress
shown (waiting for the card, n ahead; making it, the node and percent),
and the mesh in a viewer — model-viewer 4.4.0 (Apache 2.0, with three.js
and lit), served from the gateway since the page's policy loads scripts
from nowhere else — with a download and the model's licence. The mesh is
served to its owner from /3d/jobs/{id}/file; another project's person is
told there is no such job.
A job made from the portal acted for a key that does not exist: the
worker now rebuilds a session's principal from the account that made it,
while it is active and still in the project the job is charged to.
ctl 3d list|get|cancel, over /api/v1/client-jobs (/api/v1/jobs is the
control plane's own work, which ctl jobs shows).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
TripoSR's checkpoint loads with transformers 4, and ComfyUI runs 5: of a
package a node names, its range decides, and of everything else what
ComfyUI runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
G2: the gate keeps a websocket open to its ComfyUI under a client id of its
own, puts that id on every workflow it lets through — ComfyUI tells only
the client that queued a workflow how it goes — and keeps the last word on
each: the node running, its step of how many. It answers at
/progress/{prompt_id}. While a job's request waits, the router reads the
queue (queued, n ahead) and the gate (running, node, step, percent) and
writes it into the job, which a client reads back. Only into a job of the
same key: a tag naming another's job writes nothing.
G3, the rest of the plumbing: comfyui.nodes names custom nodes by https
repository and full commit, with the PyPI packages they need. The gate puts
each in the replica's own custom_nodes at that commit and installs the
packages beside the replica, constrained to the versions ComfyUI's
environment already has, which it never changes. The catalog refuses a
branch for a commit and anything but package specs for requirements. An
OBJ a node saves leaves the gateway as a GLB, colours kept, so every mesh
can carry its licence.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
G3, the part that needs no model: a ComfyUI model's comfyui.output (or
external.output) says pictures or meshes. A request for a 3D model — a
job at /v1/3d/generations — carries the picture as base64; at the replica
it is uploaded to ComfyUI's input directory (the gate lets /upload/image
through), the workflow is filled in with its name, the seed and the
resolution, and the mesh the workflow saves comes back with its type,
which is the file the job keeps. comfy.Client reads every list of files an
output names, not "images" only, and types a file by its extension where
ComfyUI sends octet-stream.
A model's license (name, url, notice) is said in /v1/models and in the
job, and written into a GLB's asset.extras, since the file travels without
the API; a mesh it cannot be written into is refused. per_mesh in
auth.pricing, usage.meshes in the rows, the sums and the export
(migration 0045). A picture asked of a mesh model, or a mesh of a picture
model, is refused saying which it makes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The catalog had one source a model, so an embedding model served by vLLM
from its repository on a card and by llama.cpp from a GGUF on CPUs had its
entry swapped by hand halfway through the metallurgy import. A profile's
source is what its engine loads and placement looks for; the model's is
the default, and what updates and canaries move. ctl model add checks
every source, the GGUF check reads each llama.cpp profile's, and ctl
weights collects none a profile loads. An empty source leaves a profile's
fingerprint as it was, so running replicas do not read as stale.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The five-minute, hourly and daily sums gain images (migration 0042), and
the CSV and JSON exports a column for them: once request rows were past
their retention, an export said what pictures cost but not how many there
were. The pictures made before are added to the sums from their rows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A control plane that has just started or become leader has heard no
requests, and read the on-demand replicas its nodes report as wanted by
nobody: it drained every one of them at every failover. An adopted replica
of an on-demand model now gets a whole idle timeout from its adoption.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model whose profiles run engine comfyui is placed on a node's card like
any other, and makes pictures there. ComfyUI asks for no key, so the agent
starts it behind llmgw comfyui-gate: ComfyUI on the loopback, the gate on
the replica's port demanding the replica's key and letting through only
what the router asks — queueing, waiting, fetching, cancelling. The gate
answers the agent's /health and /v1/models, shows ComfyUI the fetched
weights by their own names, and gives each replica its own base directory.
ComfyUI runs as the gate's child, so its card memory is the replica's.
A node offers the engine when agent.yaml names its ComfyUI
(engines.comfyui.dir). The model's comfyui: block names the workflow and,
for a source that is a set of files, the checkpoint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
auth.pricing takes per_image, charged on top of a prompt's tokens; a model
priced only per picture is not free, so a spent budget still stops it.
Each usage row keeps how many pictures were made (migration 0041), and
ctl request and the admin API's request view show them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An external model with api: comfyui is a ComfyUI server. A request for
pictures goes through the same key, allowlist, limit, admission and guard
checks as any other; at the replica it becomes a workflow — a built-in
(flux-schnell, flux-dev) or a JSON file in API format, with {{prompt}},
{{width}}, {{seed}} and the like filled in — queued, waited on, and its
pictures sent back as b64_json. A client that leaves takes its workflow
off the queue or interrupts it. ComfyUI is never scraped for metrics.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Qwen3-Embedding and models like it are trained with an instruction on the
query side only. The prefix goes before the question a knowledge-base
search embeds, at answer time and in the admin API's search; documents
reach /v1/embeddings from llmgw-kb as they are.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
What llmgw org, keys and users did against the store on a control plane
can now be done remotely and by automation: the tree read, added to,
renamed, moved and emptied; keys issued (the key in the answer, once),
listed and revoked, with the scopes' lifetime limits; people put in and
taken out of the tree, their keys in projects they no longer reach
revoked with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An adapter now has an owner (org, team, project, user or key): only callers
there see it, and to anyone else it is a model that does not exist, absent
from /v1/models too. A training job's owner defaults to the one collection
its pairs come from, must be able to read every collection they come from,
and pairs from a knowledge base are never made public. ctl train publish
keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and
sets the owned adapter after the usual architecture check.
The knowledge bases' cache_salt follows the collections searched rather
than the organization, so two teams of one organization no longer share
cached passages.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A job names the catalog model the adapter is for, the safetensors weights to
train on, the node and the MiB of a card it needs, and question/answer pairs.
The control plane sends it when the node's agent is there and nothing else
trains on it; the agent trains with its installed vLLM's torch and a pinned
overlay of peft and accelerate (no second environment), as the engines'
user, on the card with that much free — beside an engine, if it leaves that
much — converts the adapter to GGUF with the installed llama.cpp's own
conversion code when the model is served by llama.cpp, and sends what it
made back in pieces, kept in the store to download. Progress, loss and peak
card memory are reported as it goes; a node that goes fails its job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A chat model of the cluster reads each chunk and writes questions a person
might ask of it with the answers it holds, streamed as NDJSON a chunk at a
time, for people to correct before an adapter is trained on it. A chunk that
gave no pairs says so on its own line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
agents.default and agents.nodes.<id> in the cluster's settings hold
everything agent.yaml can say but the agent's id, control planes, address and
certificate, which stay in the file and are refused there. The control plane
sends each node its settings on register and, a second apart, when they
change; an agent whose settings changed checks they load beside its file,
keeps them and starts again on them, engines serving throughout. An edited
agent.yaml that loads restarts the agent too. History, audit, ctl config, the
console and the portal's settings page cover them as the rest of the
settings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On an organization, team, project or key, the narrowest winning: llmgw ctl
kb default, POST /api/v1/kb/defaults, and the console's Knowledge screen. A
request's own X-Llmgw-Knowledge overrides it and none turns it off. A default
never refuses a request: a collection the caller cannot read is left out, and
a failed search answers without passages. Removing a collection takes it out
of every default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fused by rank, a full-text match on a question's everyday words outvoted a
far nearer meaning: on the PoC, "how much leave do fathers get" found the
annual-leave page before the paternity one (cosine distance 0.44 against
0.25). Now a chunk scores its similarity to the question plus up to 0.3 for
a full-text match, each search brings four times k candidates, the text
search drops Turkish and English stop words, and an operator's search says
when the embedding model did not answer rather than quietly searching text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collections made for an organization, team, project or person, their
dimension asked of the embedding model; granted and revoked by path, never
outside their organization; their documents listed and removed; and a search
an operator can try. In the OpenAPI document.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
judge ask showed Qwen3Guard's controversial verdict as "safe (Violent)"; it
now says controversial, which a policy's judge.controversial counts. The
stray-judge note called a judge no policy asks any more "served as a model
clients use"; a model with role: guard is now said to be a judge no policy
asks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The replica was stopped and the pass went on to the other models before
placing its replacement, so a model waiting for room took the freed cards.
On the PoC a GPU judge waiting for the only card took it from
qwen3.8-35b-a3b-distill, restarted there after its definition changed,
until the judge was stopped by hand. With the waiting model first in the
catalog it happened one pass later. The replacement is now placed in the
same step that stops the old replica.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.
- format qwen3guard: its prompt written out as its template writes it, for
requests and answers, and its Safe / Unsafe / Controversial verdict read;
controversial counts as the policy's judge.controversial says, safe by
default.
- judge.models: judges asked in turn, the first that answers deciding; a
failure is recorded against the judge that failed. guard judge use sets
the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
enable serves any of them, --gpu the 4B on the roomiest GPU class as a
standard model; the leader fetches and serves one a policy names that the
catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
rather than overflowing it and refusing the request under fail: closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Issue #3 asked for commands to stop and start the system, and 05aa21c gave
only one machine's services. Now:
- llmgw stop sets serving.stopped in the cluster's settings, live: the
reconciler wants every model at no replicas, freeing every GPU, and the
gateway answers every POST under /v1/ with 503 system_stopped. The control
planes, agents and mirror keep running, and no model's definition changes.
- llmgw start clears it; every model is placed again as it asks.
- llmgw restart has every agent restart over its stream (RestartAgent), then
the other control planes over ssh, standbys before a leader, then this
one; engines keep serving.
- llmgw status is the cluster's summary, saying when it is stopped.
One machine's services moved to llmgw service status|start|stop|restart.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Checked against the installed llama.cpp (b10991): at its default verbosity
it prints neither the devices it uses nor the layers it offloads, so the
start-up output said nothing and every replica's ON column stayed empty.
The agent now takes compute from nvidia-smi - the GPUs the shard's processes
hold memory on, and how much - or CPU, for every engine, falling back to it
when llama.cpp's output says nothing; adopted engines get it too, and each
session reports it once per shard as a compute-only status. engine-check
runs its GPU token at -lv 4, where llama.cpp does say where its layers went.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It was sent once, with READY, and kept only in the leader's memory: a
failover or an agent restart lost it. The agent now keeps it with the
shard's state on disk (ShardSpec.Compute, set by Supervisor.SetCompute) and
reports it with each running shard when it registers (RunningShard.compute),
which adoption carries onto the replica.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The settling window ran from process start, so a standby that took over
after a minute placed at once, before the agents reconnected and reported
what they run: it started second copies of replicas it did not know of and
drained the healthy ones when they were adopted. Seen on the PoC: a leader
restart replaced llama-guard-3-1b and a qwen2.5-1.5b replica. Reconcile now
reopens the window on the first pass after standing by.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
