Install vLLM on every GPU node that can run it, as llama.cpp is
vLLM was installed by hand and named in each agent's configuration. The agent now installs it: a Python 3.12 environment built by uv, which is itself a GitHub release checked against its checksum, from PyPI resolved as it stood a day after that vLLM was published and installed with --require-hashes. The resolved list is kept in the environment. The environment is built relocatable under a scratch name and renamed into place once it runs, like an archive. The default is 0.30.0, whose flags were checked against the ones llmgw builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or newer, and 25 GB free while it installs; a node lacking one says which, and a node with no GPU skips it without being shown as a fault. gcc comes with it because Triton compiles its launcher at the first model load. SGLang installs the same way when the control plane names a version. llmgw doctor reports whether vLLM can be installed on the node. Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e751c89937
commit
4c5a2fd6da
21 files changed
+853
-100
No files matched your search
@@ -292,6 +292,12 @@ footer a{color:var(--muted)}
|
||||
<div class="fact"><h3>A mirror you browse {{if .Site}}<span class="next">next release</span>{{end}}</h3>
|
||||
<p>Explore what is trending on Hugging Face, read a model card under its row, and download
|
||||
once for the whole cluster. Only what your nodes can run is shown by default.</p></div>
|
||||
<div class="fact"><h3>Nodes install their own engines{{if .Site}} <span class="next">next release</span>{{end}}</h3>
|
||||
<p>llama.cpp on every node; vLLM on every GPU node that can run it, from PyPI with every
|
||||
file hash-checked. A node that cannot run it says why.</p></div>
|
||||
<div class="fact"><h3>Upgrades that keep models running</h3>
|
||||
<p>Upgrade the control plane, then the agents. A restarted agent adopts the engines it was
|
||||
running instead of reloading them, so replicas stay up throughout.</p></div>
|
||||
<div class="fact"><h3>A dashboard with nothing to set up</h3>
|
||||
<p><code>/admin</code> shows capacity per GPU class, CPU and memory per node, who is using
|
||||
what, each node's page and logs. It fits a phone. Prometheus is optional.</p></div>
|
||||
@@ -495,7 +501,6 @@ curl https://llm.example.com/v1/chat/completions \
|
||||
<code>sminfo</code>, <code>ibnetdiscover</code> and <code>nvidia-smi</code>, but not yet run
|
||||
on the hardware. Run <code>llmgw doctor --json</code> on a few nodes first.</li>
|
||||
<li>Two control planes over long periods</li>
|
||||
<li>vLLM and SGLang are not installed by the node yet; name their paths yourself</li>
|
||||
<li>Nodes do not share downloaded weights; the mirror keeps each download local</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
Reference in new issue
Block a user