Install vLLM on every GPU node that can run it, as llama.cpp is

vLLM was installed by hand and named in each agent's configuration. The
agent now installs it: a Python 3.12 environment built by uv, which is
itself a GitHub release checked against its checksum, from PyPI resolved as
it stood a day after that vLLM was published and installed with
--require-hashes. The resolved list is kept in the environment. The
environment is built relocatable under a scratch name and renamed into place
once it runs, like an archive.

The default is 0.30.0, whose flags were checked against the ones llmgw
builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or
newer, and 25 GB free while it installs; a node lacking one says which, and a
node with no GPU skips it without being shown as a fault. gcc comes with it
because Triton compiles its launcher at the first model load. SGLang
installs the same way when the control plane names a version. llmgw doctor
reports whether vLLM can be installed on the node.

Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for
CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-29 13:15:54 +02:00
1 parent e751c89937
commit 4c5a2fd6da
21 files changed
+853 -100

No files matched your search

+38 -25
View File
@@ -359,36 +359,48 @@ and gives it back when the port returns.
fabric pages show, and what to look at when a replica spanning nodes will not
run. Run `llmgw doctor` on the node before it joins.
### An engine
### Engines
Pick what you will actually serve. For a first cluster, llama.cpp is the least
trouble and runs without a GPU.
Nothing to do: the agent installs them once the node joins, under
`/opt/llmgw/engines`, and checks each one runs before the node offers it.
```
# llama.cpp, from a release build
sudo apt-get install -y libgomp1 # it will not start without this
sudo mkdir -p /opt/llama.cpp
# unpack a llama.cpp release into /opt/llama.cpp
/opt/llama.cpp/llama-server --version
| Engine | On | How |
|---|---|---|
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
vLLM is built for CUDA 13. A V100 cannot run it;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
A node with no GPU skips vLLM and says nothing about it. A node that has a GPU
but cannot run it — an old driver, a V100, a small disk — says why on the
cluster page. `llmgw doctor` checks the same before the node joins.
The versions are the control plane's, in `server.yaml`:
```yaml
engines:
vllm: "0.30.0" # the default; "" leaves vLLM to you
sglang: "0.5.20" # SGLang only when named
```
That `libgomp1` line is not padding. A `llama-server` binary missing it exists,
is executable, and cannot start — and the node reports `ready` throughout.
A changed version reaches every node when it next connects. To use an engine
you installed yourself instead, set its version to `""` and name its path in
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
For vLLM, a virtualenv on the node:
After changing a version, check it still takes the flags llmgw builds for it,
which drift between releases:
```
sudo apt-get install -y python3-venv
sudo python3 -m venv /opt/venvs/vllm
sudo /opt/venvs/vllm/bin/pip install vllm
/opt/venvs/vllm/bin/vllm --version
```
Check the engine still takes the flags llmgw builds for it, which drift between
releases:
```
llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
```
### Weights
@@ -1059,8 +1071,9 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **Engines are yours to install.** Onboarding verifies one runs; it installs
nothing.
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
install a vLLM built for your CUDA yourself.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
+14 -23
View File
@@ -116,32 +116,23 @@ The trade is live migration: a VM pinned to the host CPU cannot move to a
machine with a different one. For a node whose job is arithmetic that is the
right way round, and `--cpu` overrides it where it is not.
## What onboarding does not do
## Engines
It gives a machine an identity and an agent. It does **not** install inference
engines or model weights, and a node missing either looks healthy in
`llmgw ctl nodes` right up until a replica is placed on it and the shard dies
with "executable file not found".
The script installs none, and does not need to: once the agent is connected,
the control plane tells it which engines to have and it installs them itself
— llama.cpp on every node, vLLM on every node with an NVIDIA GPU it can run
on, SGLang where the control plane names a version. Each is checked to run
before the node offers it, so a node never advertises an engine that cannot
start. [install.md](install.md#engines) says what vLLM needs of the node.
Onboarding does check that an engine you point it at can be executed, and says
so loudly when it cannot. That check exists because the failure is invisible
otherwise: `llama-server` on node05 was present, executable, and could not start
for want of `libgomp1`, and the node reported `ready` in `llmgw ctl nodes` the
whole time. Being told at onboarding beats finding out when a replica is placed
and the reason is three logs away.
`--llamacpp-bin`, `--vllm-bin` and `--sglang-python` are for an engine you
installed yourself. For the agent to use yours rather than install its own,
also set that engine's version to `""` under `engines` in `server.yaml`.
So the node is not finished until:
- the engines its catalog profiles name are installed (`vllm`, `sglang`,
`llama-server`), at the paths the agent configuration gives
- the weights are on disk, at the `source` each model declares
Both are deliberately outside this script. Engines are a platform decision with
versions and drivers attached, and distributing weights is its own problem —
hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
a laptop. llmgw will fetch weights for a model whose `source` is a URL, one
copy per node, which covers the simple case; anything larger stays the
operator's arrangement, and llmgw is content to find the files already there.
Weights are the other thing a node needs. llmgw fetches them for a model whose
`source` is a URL, one copy per node, which covers the simple case; anything
larger — a shared filesystem, a cache — stays the operator's arrangement, and
llmgw is content to find the files already there.
The `sim` engine needs neither, which is what makes it useful for exercising
everything else.