Launch multi-node vLLM its own way by default, not through Ray
An empty launch_mode meant ray: a Ray head and joins on every node, a dependency to install, and a path never checked against a real cluster. It now means mp - every node runs vllm serve with its rank and the leader's address, the workers headless - which is how vLLM documents a multi-node launch today and needs nothing but vLLM. ray is still there for a vLLM too old for mp's flags. The example agent configuration, doctor and the reference say so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
20479a2d5f
commit
45e3f52da3
5 files changed
+34
-16
No files matched your search
@@ -96,10 +96,11 @@ discovery:
|
||||
engines:
|
||||
vllm:
|
||||
bin: /opt/venvs/vllm/bin/vllm
|
||||
# Which multi-node mechanism works depends on the vLLM version, and the
|
||||
# flags have changed more than once. Check the version you deploy.
|
||||
launch_mode: ray # "ray" or "mp"
|
||||
ray_bin: /opt/venvs/vllm/bin/ray
|
||||
# How a replica spanning nodes is launched. mp, the default, is what vLLM
|
||||
# documents today and needs nothing but vLLM; ray is for a vLLM too old
|
||||
# for mp's multi-node flags, and needs Ray on every node.
|
||||
launch_mode: mp # "mp" or "ray"
|
||||
# ray_bin: /opt/venvs/vllm/bin/ray
|
||||
sglang:
|
||||
# SGLang is launched as a module, which is its documented entry point.
|
||||
python: /opt/venvs/sglang/bin/python
|
||||
|
||||
Reference in new issue
Block a user