Install vLLM on every GPU node that can run it, as llama.cpp is
vLLM was installed by hand and named in each agent's configuration. The agent now installs it: a Python 3.12 environment built by uv, which is itself a GitHub release checked against its checksum, from PyPI resolved as it stood a day after that vLLM was published and installed with --require-hashes. The resolved list is kept in the environment. The environment is built relocatable under a scratch name and renamed into place once it runs, like an archive. The default is 0.30.0, whose flags were checked against the ones llmgw builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or newer, and 25 GB free while it installs; a node lacking one says which, and a node with no GPU skips it without being shown as a fault. gcc comes with it because Triton compiles its launcher at the first model load. SGLang installs the same way when the control plane names a version. llmgw doctor reports whether vLLM can be installed on the node. Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e751c89937
commit
4c5a2fd6da
21 files changed
+853
-100
No files matched your search
+14
-23
@@ -116,32 +116,23 @@ The trade is live migration: a VM pinned to the host CPU cannot move to a
|
||||
machine with a different one. For a node whose job is arithmetic that is the
|
||||
right way round, and `--cpu` overrides it where it is not.
|
||||
|
||||
## What onboarding does not do
|
||||
## Engines
|
||||
|
||||
It gives a machine an identity and an agent. It does **not** install inference
|
||||
engines or model weights, and a node missing either looks healthy in
|
||||
`llmgw ctl nodes` right up until a replica is placed on it and the shard dies
|
||||
with "executable file not found".
|
||||
The script installs none, and does not need to: once the agent is connected,
|
||||
the control plane tells it which engines to have and it installs them itself
|
||||
— llama.cpp on every node, vLLM on every node with an NVIDIA GPU it can run
|
||||
on, SGLang where the control plane names a version. Each is checked to run
|
||||
before the node offers it, so a node never advertises an engine that cannot
|
||||
start. [install.md](install.md#engines) says what vLLM needs of the node.
|
||||
|
||||
Onboarding does check that an engine you point it at can be executed, and says
|
||||
so loudly when it cannot. That check exists because the failure is invisible
|
||||
otherwise: `llama-server` on node05 was present, executable, and could not start
|
||||
for want of `libgomp1`, and the node reported `ready` in `llmgw ctl nodes` the
|
||||
whole time. Being told at onboarding beats finding out when a replica is placed
|
||||
and the reason is three logs away.
|
||||
`--llamacpp-bin`, `--vllm-bin` and `--sglang-python` are for an engine you
|
||||
installed yourself. For the agent to use yours rather than install its own,
|
||||
also set that engine's version to `""` under `engines` in `server.yaml`.
|
||||
|
||||
So the node is not finished until:
|
||||
|
||||
- the engines its catalog profiles name are installed (`vllm`, `sglang`,
|
||||
`llama-server`), at the paths the agent configuration gives
|
||||
- the weights are on disk, at the `source` each model declares
|
||||
|
||||
Both are deliberately outside this script. Engines are a platform decision with
|
||||
versions and drivers attached, and distributing weights is its own problem —
|
||||
hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
|
||||
a laptop. llmgw will fetch weights for a model whose `source` is a URL, one
|
||||
copy per node, which covers the simple case; anything larger stays the
|
||||
operator's arrangement, and llmgw is content to find the files already there.
|
||||
Weights are the other thing a node needs. llmgw fetches them for a model whose
|
||||
`source` is a URL, one copy per node, which covers the simple case; anything
|
||||
larger — a shared filesystem, a cache — stays the operator's arrangement, and
|
||||
llmgw is content to find the files already there.
|
||||
|
||||
The `sim` engine needs neither, which is what makes it useful for exercising
|
||||
everything else.
|
||||
|
||||
Reference in new issue
Block a user