Written for whoever is standing up the cluster with root on some servers and nothing installed. Control plane, certificate authority, first node, first model, first request. It leads with what llmgw does not install — drivers, CUDA, engines, weights — because a node missing any of them registers, reports ready, and fails every replica placed on it, and the reason is three logs away. The useful order is to make a node able to serve before onboarding it, which is the opposite of what the tooling's convenience suggests. The failures it warns about are ones this cluster actually hit: llama-server present and unable to start for want of libgomp1 while the node reported healthy; a Proxmox VM on the default kvm64 CPU, where inference is six to nine times slower and nothing looks broken because every number is simply small; requests dying at exactly sixty seconds on a liveness timeout that is a generation budget in disguise for non-streaming calls. Verified rather than recalled. Every command it prints was checked against the binary, every config key against the struct tags, and its own verification and troubleshooting commands were run against the live cluster. That caught two errors in my draft: the health endpoint is /health and returns an empty body, not /healthz, and the Go version needed is 1.26. The known-gaps section is there so a reader finds them on a Tuesday afternoon rather than at 2am: weights and engines are theirs to distribute, there is one control plane and no HA, and multi-node replicas have never run on real InfiniBand — only against simulated hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d07dddb1bb
commit
ea4dd37f80
2 files changed
+484
No files matched your search
@@ -6,6 +6,18 @@ heterogeneous GPUs.
|
||||
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
||||
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
||||
|
||||
## Getting started
|
||||
|
||||
Starting from nothing — no packages installed, no drivers, no nodes — follow
|
||||
[docs/install.md](docs/install.md): it builds a control plane, adds a node and
|
||||
serves a model, and says plainly which parts llmgw does not install for you.
|
||||
|
||||
| Guide | For |
|
||||
|---|---|
|
||||
| [docs/install.md](docs/install.md) | Standing up a cluster from scratch |
|
||||
| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists |
|
||||
| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys |
|
||||
|
||||
## Status
|
||||
|
||||
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
|
||||
Reference in new issue
Block a user