Install vLLM on every GPU node that can run it, as llama.cpp is
vLLM was installed by hand and named in each agent's configuration. The agent now installs it: a Python 3.12 environment built by uv, which is itself a GitHub release checked against its checksum, from PyPI resolved as it stood a day after that vLLM was published and installed with --require-hashes. The resolved list is kept in the environment. The environment is built relocatable under a scratch name and renamed into place once it runs, like an archive. The default is 0.30.0, whose flags were checked against the ones llmgw builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or newer, and 25 GB free while it installs; a node lacking one says which, and a node with no GPU skips it without being shown as a fault. gcc comes with it because Triton compiles its launcher at the first model load. SGLang installs the same way when the control plane names a version. llmgw doctor reports whether vLLM can be installed on the node. Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e751c89937
commit
4c5a2fd6da
21 files changed
+853
-100
No files matched your search
+38
-25
@@ -359,36 +359,48 @@ and gives it back when the port returns.
|
||||
fabric pages show, and what to look at when a replica spanning nodes will not
|
||||
run. Run `llmgw doctor` on the node before it joins.
|
||||
|
||||
### An engine
|
||||
### Engines
|
||||
|
||||
Pick what you will actually serve. For a first cluster, llama.cpp is the least
|
||||
trouble and runs without a GPU.
|
||||
Nothing to do: the agent installs them once the node joins, under
|
||||
`/opt/llmgw/engines`, and checks each one runs before the node offers it.
|
||||
|
||||
```
|
||||
# llama.cpp, from a release build
|
||||
sudo apt-get install -y libgomp1 # it will not start without this
|
||||
sudo mkdir -p /opt/llama.cpp
|
||||
# unpack a llama.cpp release into /opt/llama.cpp
|
||||
/opt/llama.cpp/llama-server --version
|
||||
| Engine | On | How |
|
||||
|---|---|---|
|
||||
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
|
||||
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
|
||||
node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
|
||||
vLLM is built for CUDA 13. A V100 cannot run it;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
A node with no GPU skips vLLM and says nothing about it. A node that has a GPU
|
||||
but cannot run it — an old driver, a V100, a small disk — says why on the
|
||||
cluster page. `llmgw doctor` checks the same before the node joins.
|
||||
|
||||
The versions are the control plane's, in `server.yaml`:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
vllm: "0.30.0" # the default; "" leaves vLLM to you
|
||||
sglang: "0.5.20" # SGLang only when named
|
||||
```
|
||||
|
||||
That `libgomp1` line is not padding. A `llama-server` binary missing it exists,
|
||||
is executable, and cannot start — and the node reports `ready` throughout.
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself instead, set its version to `""` and name its path in
|
||||
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
|
||||
|
||||
For vLLM, a virtualenv on the node:
|
||||
After changing a version, check it still takes the flags llmgw builds for it,
|
||||
which drift between releases:
|
||||
|
||||
```
|
||||
sudo apt-get install -y python3-venv
|
||||
sudo python3 -m venv /opt/venvs/vllm
|
||||
sudo /opt/venvs/vllm/bin/pip install vllm
|
||||
/opt/venvs/vllm/bin/vllm --version
|
||||
```
|
||||
|
||||
Check the engine still takes the flags llmgw builds for it, which drift between
|
||||
releases:
|
||||
|
||||
```
|
||||
llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
|
||||
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
|
||||
```
|
||||
|
||||
### Weights
|
||||
@@ -1059,8 +1071,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **Engines are yours to install.** Onboarding verifies one runs; it installs
|
||||
nothing.
|
||||
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
|
||||
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
|
||||
install a vLLM built for your CUDA yourself.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
Reference in new issue
Block a user