Serve vision models with their projector and original weights on vLLM; group the mirror by version

model serve takes --projector, fetched beside the weights and loaded by
llama.cpp with --mmproj, and --format safetensors with --gpu-class and --tp,
which fetches the weights with their config and tokenizer and runs them on
vLLM. The console offers both, F switching between GGUF and original
weights. The mirror view is one row per version with its files on Enter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 00:42:59 +02:00
1 parent 7aefb236d6
commit a16097cfe8
12 files changed
+509 -81

No files matched your search

+19 -9
View File
@@ -111,9 +111,13 @@ which): requests, tokens, errors and spend.
team or project is guaranteed and may at most hold, and what it holds now.
**A** adds one; on a row, change or remove it.
**Mirror** is what the model mirror holds, how far each download has got, and
which models use each file. **A** downloads a URL into it; on a row, where
nodes fetch it from, or remove it — with a warning when a model still uses it.
**Mirror** is what the model mirror holds, one row for each version of each
model — a file, or the set of files a model of several is: shards, a vision
projector, a config and tokenizer — with how far its download has got and
which models use it. **A** downloads a URL into it; on a row, its files with
their checksums and where each was fetched from, the source a model
definition names for it, or remove it — with a warning when a model still
uses it.
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
coloured.
@@ -181,12 +185,18 @@ to restart it. Another control plane keeps its own copy.
certificate and starts its agent (`sudo llmgw node add`).
- **Replicate the store** — moves the store onto two more machines under
Patroni (offered only while it is not replicated).
- **Serve a model** — Hugging Face's GGUF models: trending, most downloaded,
most liked or newest (**S** changes which), or **/** to search. Enter on
one lists its files with their sizes; the one chosen is fetched into the
mirror, checked, added to the catalog and started, and a replica is asked
to say hello (`llmgw ctl model serve`). `mirror.registry` points it at
another registry, for a site without the internet.
- **Serve a model** — Hugging Face's models: trending, most downloaded,
most liked or newest (**S** changes which), or **/** to search. **F**
switches between GGUF files, which llama.cpp runs on the CPU or a GPU, and
original safetensors weights, which vLLM runs on GPUs. Enter on a GGUF
repository lists its files with their sizes; a vision model offers its
projector too, which is fetched beside it and loaded with it so the model
reads images. Enter on original weights offers them with their config and
tokenizer, a GPU class and how many GPUs a replica spans. What is chosen is
fetched into the mirror, checked, added to the catalog and started, and a
replica is asked to say hello (`llmgw ctl model serve`).
`mirror.registry` points it at another registry, for a site without the
internet.
- **Upgrade the cluster** to a release: this machine, then every node in turn
(`sudo llmgw ctl upgrade`).
- **Restart all replicas**, one at a time.