The console starts a serve or update job over the admin API and stays,
instead of closing to follow the download. Jobs lists them with their
progress and steps; the sidebar counts those running. ctl model serve and
update take --detach; ctl jobs lists jobs and follows one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ranking what was free first made every file come out on the CPU on a cluster
whose one card was in use. The plan now ranks by hardware alone, free now
only breaking a tie; a model served onto cards that are busy is added and
waits for them, the job ending as waiting rather than failed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane reads the model's shape from one GGUF header or its
config.json and lists, for every file, the best way it runs here: one card,
several of a node's (by layer for llama.cpp, tensor-parallel for vLLM),
several nodes' over one fabric (vLLM, pipeline-parallel between them), or a
CPU node; what each card holds and whether it is free now. ctl model plan
prints it; model serve takes its file and way unless told, gives a GGUF on
GPUs a GPU profile, and with --fallbacks adds the next best ways. The
console shows it per file and asks for the way and the fallback.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
model serve takes --projector, fetched beside the weights and loaded by
llama.cpp with --mmproj, and --format safetensors with --gpu-class and --tp,
which fetches the weights with their config and tokenizer and runs them on
vLLM. The console offers both, F switching between GGUF and original
weights. The mirror view is one row per version with its files on Enter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The mirror keeps where it fetched a file beside it, and reports that and the
checksum after it restarts. A model whose files the mirror cannot trace is
found on the registry: a repository holding a file of that name that has the
commit the mirror's directory is named after, with every file at the same
size, the most downloaded first. ctl model origin and the console set one by
hand, checked the same way. Commits compare shortened.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model served from the registry records its origin: repository, commit and
each file's checksum; one served before is worked out from the mirror's
record of where it fetched the file. Every mirror.update_check (6h) the
control plane asks whether those files changed there. ctl model updates
lists it, ctl model update fetches beside the old files, checks, swaps and
replaces replicas one at a time, ctl model rollback goes back. The console
shows it on Models and offers the same.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A profile's cpu_cores is reserved out of its node's CPUs by placement, as
memory is, and becomes the engine's thread count. Agents report their CPU
count in registration and heartbeats. New CPU models must declare memory;
model views show reserved against measured memory with warnings, and
ctl model resources sets both for a stored model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adding a model took four steps and a YAML file — plan it, fetch it into the
mirror, write its catalog entry with a memory figure somebody estimated,
scale it — and whether it would load was found out at the end.
`llmgw ctl model serve OWNER/REPO` does it as one job in the control plane,
where the registry, the mirror, the catalog and the fleet already are:
- the registry: the GGUF file set matching --quant (Q4_K_M by default) or
--file, leaving out vision projectors and importance matrices, and naming
what the repository does have when nothing matches; gated repositories are
refused;
- the mirror: fetched at the repository's revision, with progress, and each
file's sha256 compared with what the registry published — the mirror is
the one place a corrupted download is caught once rather than loaded on
every node;
- the check: whether the nodes' llama.cpp loads the architecture, and the
memory it needs from its own header;
- the catalog, then a wait for a replica that stops early, with the reason,
when the model is parked or fits on no node;
- a smoke test: one short request to the replica, with its engine key, and
the answer and tokens per second.
ctl starts the job and prints each step as it changes; a download of hours
does not depend on the terminal. --on-demand adds the model without starting
a replica. Jobs are kept in memory until the control plane restarts.
The catalog name a repository suggests moved into the registry package, so
the mirror page and this agree on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
