Add a quick install guide: six steps from empty servers to a model
install.md explains every choice and is long for someone who wants the commands. quick-install.md has only those, and what to check after each: the binary, server init, onboarding GPU nodes (with doctor first and the script fetched from the release tag, since it is not installed anywhere), a model from the mirror's explorer, keys and editors, and Caddy. Every button and path it names was checked against the pages. quickstart.md is now said to be the from-source, one-workstation path it is; README and the about page point at the new guide. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed
1 parent
4c5a2fd6da
commit
c8da06988f
5 files changed
+198
-8
No files matched your search
@@ -58,9 +58,11 @@ them. On such a machine, run `llmgw doctor --json` first and keep the report;
|
||||
|
||||
## Documentation
|
||||
|
||||
- [docs/install.md](docs/install.md) — installing from nothing: control plane,
|
||||
nodes, engines, weights, first request
|
||||
- [docs/quickstart.md](docs/quickstart.md) — the short path
|
||||
- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in
|
||||
six steps: start here
|
||||
- [docs/install.md](docs/install.md) — the same from nothing, with every option
|
||||
explained: control plane, nodes, engines, weights, first request
|
||||
- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source
|
||||
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
||||
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
||||
when a replica spanning nodes will not run
|
||||
|
||||
+2
-1
@@ -1,7 +1,8 @@
|
||||
# Installing llmgw from nothing
|
||||
|
||||
Written for whoever is standing up the cluster: you have servers, you have root
|
||||
on them, and nothing is installed yet. By the end there is a control plane, a
|
||||
on them, and nothing is installed yet. For only the commands, read
|
||||
[quick-install.md](quick-install.md) first; this page explains them. By the end there is a control plane, a
|
||||
node in the fleet, and a model answering requests.
|
||||
|
||||
It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
# Quick install
|
||||
|
||||
From empty servers to a model answering requests, in six steps. This page
|
||||
has only the commands and what to check after each; [install.md](install.md)
|
||||
explains every choice.
|
||||
|
||||
## What you need
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
|
||||
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
|
||||
| **root** | on the control plane, and `sudo` without a password on every node. |
|
||||
| **SSH** | from root on the control plane to each node. |
|
||||
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
|
||||
|
||||
You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
|
||||
themselves once they join.
|
||||
|
||||
## 1. Put llmgw on the control plane
|
||||
|
||||
```
|
||||
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
|
||||
| grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
|
||||
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
|
||||
curl -fsSLO $U/llmgw-$VERSION-linux-amd64 # -linux-arm64 on ARM
|
||||
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
|
||||
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
|
||||
llmgw --version
|
||||
```
|
||||
|
||||
`sha256sum` must print `OK`. If it does not, the download is damaged; do not
|
||||
install it.
|
||||
|
||||
## 2. Set up the control plane
|
||||
|
||||
```
|
||||
sudo llmgw server init --admin you@example.com
|
||||
```
|
||||
|
||||
This one command sets up everything on this machine, and takes a few minutes:
|
||||
|
||||
- the service, its certificates and its configuration;
|
||||
- your administrator account. **The password is printed once. Copy it now.**
|
||||
- a model mirror, which downloads model files once for the whole cluster;
|
||||
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
|
||||
|
||||
If nodes will reach this machine by a name or address it does not know itself
|
||||
(a DNS name, a NAT address), add `--host` for each:
|
||||
|
||||
```
|
||||
sudo llmgw server init --admin you@example.com --host llm.example.com
|
||||
```
|
||||
|
||||
It is safe to run again. It keeps what exists and finishes what is missing.
|
||||
|
||||
**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
|
||||
password, and ask something. The small model answers. Change the password on
|
||||
the account page.
|
||||
|
||||
## 3. Add the GPU nodes
|
||||
|
||||
**On each node**, check it first. Set `VERSION` and `U` as in step 1, then:
|
||||
|
||||
```
|
||||
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
|
||||
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
|
||||
sudo llmgw doctor
|
||||
```
|
||||
|
||||
Fix every `FAIL` line before going on. The fix is printed under each line.
|
||||
|
||||
**On the control plane**, get the onboarding script once:
|
||||
|
||||
```
|
||||
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/raw/tag/$VERSION/deploy/bootstrap/onboard-node.sh
|
||||
chmod +x onboard-node.sh
|
||||
```
|
||||
|
||||
Check that root can reach the node over SSH. This must print `ok` without asking
|
||||
for a password:
|
||||
|
||||
```
|
||||
sudo ssh ubuntu@10.0.0.21 'sudo true && echo ok'
|
||||
```
|
||||
|
||||
Then add the node. `--node` is its name in the cluster; use its hostname.
|
||||
|
||||
```
|
||||
sudo ./onboard-node.sh --node gpu01 --addr 10.0.0.21 \
|
||||
--server https://10.0.0.10:7443 --ssh-user ubuntu
|
||||
```
|
||||
|
||||
Replace `10.0.0.21` with the node's address, `10.0.0.10` with the control
|
||||
plane's, and `ubuntu` with the user you SSH in as.
|
||||
|
||||
If the node has InfiniBand, name the fabric and its adapters, and give every
|
||||
node on the same fabric the same name:
|
||||
|
||||
```
|
||||
--fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
|
||||
```
|
||||
|
||||
**Check:**
|
||||
|
||||
```
|
||||
llmgw ctl nodes
|
||||
```
|
||||
|
||||
The node appears within seconds. It then installs its engines by itself:
|
||||
llama.cpp in about a minute, vLLM in a few more (about 8 GB). The cluster page
|
||||
(`/admin`) shows the progress under each node. If a node cannot run vLLM, it
|
||||
says why on the same page.
|
||||
|
||||
Repeat for each node.
|
||||
|
||||
## 4. Add a model
|
||||
|
||||
Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
|
||||
Hugging Face, showing only what your nodes can run.
|
||||
|
||||
1. Click a model to read its card.
|
||||
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
||||
**Fetch**. It is downloaded into the mirror once, for every node.
|
||||
3. When the download finishes, click **serve** on its row in the table above.
|
||||
4. In the form, choose how it runs: **always on**, or **on demand**, which
|
||||
loads it on the first request and stops it after 15 minutes unused. Click
|
||||
**Add to the catalog**.
|
||||
|
||||
The model is placed on a node that fits it, and its status on `/admin` turns
|
||||
`ready`.
|
||||
|
||||
**Check:** the model appears in the list on `/chat`. Ask it something.
|
||||
|
||||
## 5. Give people access
|
||||
|
||||
**A key for a program:** each person makes their own on the account page
|
||||
(`/profile`). The key is shown once.
|
||||
|
||||
```
|
||||
curl http://<control-plane>:8080/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
|
||||
```
|
||||
|
||||
Any OpenAI client works. Only the base URL changes:
|
||||
`http://<control-plane>:8080/v1`.
|
||||
|
||||
**An editor:** the account page connects Continue, Codex or Cline. Pick the
|
||||
tool and a model, and it gives a one-line command that writes the
|
||||
configuration.
|
||||
|
||||
**Other people's accounts:** they sign up at `/signup`, and you approve them on
|
||||
the people page (`/admin/users`).
|
||||
|
||||
## 6. Put it behind HTTPS
|
||||
|
||||
Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
|
||||
reverse proxy in front with a real certificate. With Caddy, which gets the
|
||||
certificate by itself:
|
||||
|
||||
```
|
||||
sudo apt-get install -y caddy # Debian and Ubuntu
|
||||
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
|
||||
sudo systemctl reload caddy
|
||||
```
|
||||
|
||||
Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
|
||||
through the proxy. There is an nginx configuration too:
|
||||
[deploy/proxy/](../deploy/proxy/).
|
||||
|
||||
## When something does not work
|
||||
|
||||
| What you see | What to do |
|
||||
|---|---|
|
||||
| A node does not appear in `llmgw ctl nodes` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443, or the control plane's address is not in its certificate: run `server init` again with `--host`. |
|
||||
| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
|
||||
| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
|
||||
| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
|
||||
| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |
|
||||
|
||||
## Next
|
||||
|
||||
- [install.md](install.md): the same steps with every option explained.
|
||||
- [fabrics.md](fabrics.md): InfiniBand and NVLink.
|
||||
- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.
|
||||
+3
-3
@@ -1,8 +1,8 @@
|
||||
# Quickstart: one machine, one model, ten minutes
|
||||
|
||||
Written for someone who wants to see llmgw answer a request on a single
|
||||
workstation before reading anything else. No systemd, no `sudo`. For a real
|
||||
cluster, see [install.md](install.md).
|
||||
Written for a developer who wants to see llmgw answer a request on a single
|
||||
workstation, built from source, with no systemd and no `sudo`. **To install
|
||||
llmgw on servers, read [quick-install.md](quick-install.md) instead.**
|
||||
|
||||
What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
|
||||
of a real model served by llama.cpp.
|
||||
|
||||
@@ -203,7 +203,7 @@ footer a{color:var(--muted)}
|
||||
fabric, and a model split across machines never straddles two.</p>
|
||||
<div class="cta">
|
||||
<a class="btn" href="#download">Download{{with .Release}} {{.}}{{end}}</a>
|
||||
<a class="btn ghost" href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/install.md">Install guide</a>
|
||||
<a class="btn ghost" href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/quick-install.md">Install guide</a>
|
||||
</div>
|
||||
<div class="oneline" aria-label="Set up a control plane">
|
||||
<code><span class="p">$ </span>sudo llmgw server init --admin you@example.com</code>
|
||||
|
||||
Reference in new issue
Block a user