Document installing from nothing
CI / test (push) Successful in 1m24s

Written for whoever is standing up the cluster with root on some servers and
nothing installed. Control plane, certificate authority, first node, first
model, first request.

It leads with what llmgw does not install — drivers, CUDA, engines, weights
— because a node missing any of them registers, reports ready, and fails
every replica placed on it, and the reason is three logs away. The useful
order is to make a node able to serve before onboarding it, which is the
opposite of what the tooling's convenience suggests.

The failures it warns about are ones this cluster actually hit: llama-server
present and unable to start for want of libgomp1 while the node reported
healthy; a Proxmox VM on the default kvm64 CPU, where inference is six to
nine times slower and nothing looks broken because every number is simply
small; requests dying at exactly sixty seconds on a liveness timeout that is
a generation budget in disguise for non-streaming calls.

Verified rather than recalled. Every command it prints was checked against
the binary, every config key against the struct tags, and its own
verification and troubleshooting commands were run against the live cluster.
That caught two errors in my draft: the health endpoint is /health and
returns an empty body, not /healthz, and the Go version needed is 1.26.

The known-gaps section is there so a reader finds them on a Tuesday
afternoon rather than at 2am: weights and engines are theirs to distribute,
there is one control plane and no HA, and multi-node replicas have never run
on real InfiniBand — only against simulated hardware.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 07:33:49 +02:00
1 parent d07dddb1bb
commit ea4dd37f80
2 files changed
+484

No files matched your search

+12
View File
@@ -6,6 +6,18 @@ heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
## Getting started
Starting from nothing — no packages installed, no drivers, no nodes — follow
[docs/install.md](docs/install.md): it builds a control plane, adds a node and
serves a model, and says plainly which parts llmgw does not install for you.
| Guide | For |
|---|---|
| [docs/install.md](docs/install.md) | Standing up a cluster from scratch |
| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists |
| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys |
## Status
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).