Serve vision models with their projector and original weights on vLLM; group the mirror by version
model serve takes --projector, fetched beside the weights and loaded by llama.cpp with --mmproj, and --format safetensors with --gpu-class and --tp, which fetches the weights with their config and tokenizer and runs them on vLLM. The console offers both, F switching between GGUF and original weights. The mirror view is one row per version with its files on Enter. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
7aefb236d6
commit
a16097cfe8
12 files changed
+509
-81
No files matched your search
+19
-9
@@ -111,9 +111,13 @@ which): requests, tokens, errors and spend.
|
||||
team or project is guaranteed and may at most hold, and what it holds now.
|
||||
**A** adds one; on a row, change or remove it.
|
||||
|
||||
**Mirror** is what the model mirror holds, how far each download has got, and
|
||||
which models use each file. **A** downloads a URL into it; on a row, where
|
||||
nodes fetch it from, or remove it — with a warning when a model still uses it.
|
||||
**Mirror** is what the model mirror holds, one row for each version of each
|
||||
model — a file, or the set of files a model of several is: shards, a vision
|
||||
projector, a config and tokenizer — with how far its download has got and
|
||||
which models use it. **A** downloads a URL into it; on a row, its files with
|
||||
their checksums and where each was fetched from, the source a model
|
||||
definition names for it, or remove it — with a warning when a model still
|
||||
uses it.
|
||||
|
||||
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
|
||||
coloured.
|
||||
@@ -181,12 +185,18 @@ to restart it. Another control plane keeps its own copy.
|
||||
certificate and starts its agent (`sudo llmgw node add`).
|
||||
- **Replicate the store** — moves the store onto two more machines under
|
||||
Patroni (offered only while it is not replicated).
|
||||
- **Serve a model** — Hugging Face's GGUF models: trending, most downloaded,
|
||||
most liked or newest (**S** changes which), or **/** to search. Enter on
|
||||
one lists its files with their sizes; the one chosen is fetched into the
|
||||
mirror, checked, added to the catalog and started, and a replica is asked
|
||||
to say hello (`llmgw ctl model serve`). `mirror.registry` points it at
|
||||
another registry, for a site without the internet.
|
||||
- **Serve a model** — Hugging Face's models: trending, most downloaded,
|
||||
most liked or newest (**S** changes which), or **/** to search. **F**
|
||||
switches between GGUF files, which llama.cpp runs on the CPU or a GPU, and
|
||||
original safetensors weights, which vLLM runs on GPUs. Enter on a GGUF
|
||||
repository lists its files with their sizes; a vision model offers its
|
||||
projector too, which is fetched beside it and loaded with it so the model
|
||||
reads images. Enter on original weights offers them with their config and
|
||||
tokenizer, a GPU class and how many GPUs a replica spans. What is chosen is
|
||||
fetched into the mirror, checked, added to the catalog and started, and a
|
||||
replica is asked to say hello (`llmgw ctl model serve`).
|
||||
`mirror.registry` points it at another registry, for a site without the
|
||||
internet.
|
||||
- **Upgrade the cluster** to a release: this machine, then every node in turn
|
||||
(`sudo llmgw ctl upgrade`).
|
||||
- **Restart all replicas**, one at a time.
|
||||
|
||||
Reference in new issue
Block a user