Add a quick install guide: six steps from empty servers to a model
install.md explains every choice and is long for someone who wants the commands. quick-install.md has only those, and what to check after each: the binary, server init, onboarding GPU nodes (with doctor first and the script fetched from the release tag, since it is not installed anywhere), a model from the mirror's explorer, keys and editors, and Caddy. Every button and path it names was checked against the pages. quickstart.md is now said to be the from-source, one-workstation path it is; README and the about page point at the new guide. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
4c5a2fd6da
commit
c8da06988f
5 files changed
+198
-8
No files matched your search
@@ -58,9 +58,11 @@ them. On such a machine, run `llmgw doctor --json` first and keep the report;
|
|||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- [docs/install.md](docs/install.md) — installing from nothing: control plane,
|
- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in
|
||||||
nodes, engines, weights, first request
|
six steps: start here
|
||||||
- [docs/quickstart.md](docs/quickstart.md) — the short path
|
- [docs/install.md](docs/install.md) — the same from nothing, with every option
|
||||||
|
explained: control plane, nodes, engines, weights, first request
|
||||||
|
- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source
|
||||||
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
||||||
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
||||||
when a replica spanning nodes will not run
|
when a replica spanning nodes will not run
|
||||||
|
|||||||
+2
-1
@@ -1,7 +1,8 @@
|
|||||||
# Installing llmgw from nothing
|
# Installing llmgw from nothing
|
||||||
|
|
||||||
Written for whoever is standing up the cluster: you have servers, you have root
|
Written for whoever is standing up the cluster: you have servers, you have root
|
||||||
on them, and nothing is installed yet. By the end there is a control plane, a
|
on them, and nothing is installed yet. For only the commands, read
|
||||||
|
[quick-install.md](quick-install.md) first; this page explains them. By the end there is a control plane, a
|
||||||
node in the fleet, and a model answering requests.
|
node in the fleet, and a model answering requests.
|
||||||
|
|
||||||
It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond
|
It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond
|
||||||
|
|||||||
@@ -0,0 +1,187 @@
|
|||||||
|
# Quick install
|
||||||
|
|
||||||
|
From empty servers to a model answering requests, in six steps. This page
|
||||||
|
has only the commands and what to check after each; [install.md](install.md)
|
||||||
|
explains every choice.
|
||||||
|
|
||||||
|
## What you need
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
|
||||||
|
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
|
||||||
|
| **root** | on the control plane, and `sudo` without a password on every node. |
|
||||||
|
| **SSH** | from root on the control plane to each node. |
|
||||||
|
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
|
||||||
|
|
||||||
|
You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
|
||||||
|
themselves once they join.
|
||||||
|
|
||||||
|
## 1. Put llmgw on the control plane
|
||||||
|
|
||||||
|
```
|
||||||
|
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
|
||||||
|
| grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
|
||||||
|
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
|
||||||
|
curl -fsSLO $U/llmgw-$VERSION-linux-amd64 # -linux-arm64 on ARM
|
||||||
|
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
|
||||||
|
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
|
||||||
|
llmgw --version
|
||||||
|
```
|
||||||
|
|
||||||
|
`sha256sum` must print `OK`. If it does not, the download is damaged; do not
|
||||||
|
install it.
|
||||||
|
|
||||||
|
## 2. Set up the control plane
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo llmgw server init --admin you@example.com
|
||||||
|
```
|
||||||
|
|
||||||
|
This one command sets up everything on this machine, and takes a few minutes:
|
||||||
|
|
||||||
|
- the service, its certificates and its configuration;
|
||||||
|
- your administrator account. **The password is printed once. Copy it now.**
|
||||||
|
- a model mirror, which downloads model files once for the whole cluster;
|
||||||
|
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
|
||||||
|
|
||||||
|
If nodes will reach this machine by a name or address it does not know itself
|
||||||
|
(a DNS name, a NAT address), add `--host` for each:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo llmgw server init --admin you@example.com --host llm.example.com
|
||||||
|
```
|
||||||
|
|
||||||
|
It is safe to run again. It keeps what exists and finishes what is missing.
|
||||||
|
|
||||||
|
**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
|
||||||
|
password, and ask something. The small model answers. Change the password on
|
||||||
|
the account page.
|
||||||
|
|
||||||
|
## 3. Add the GPU nodes
|
||||||
|
|
||||||
|
**On each node**, check it first. Set `VERSION` and `U` as in step 1, then:
|
||||||
|
|
||||||
|
```
|
||||||
|
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
|
||||||
|
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
|
||||||
|
sudo llmgw doctor
|
||||||
|
```
|
||||||
|
|
||||||
|
Fix every `FAIL` line before going on. The fix is printed under each line.
|
||||||
|
|
||||||
|
**On the control plane**, get the onboarding script once:
|
||||||
|
|
||||||
|
```
|
||||||
|
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/raw/tag/$VERSION/deploy/bootstrap/onboard-node.sh
|
||||||
|
chmod +x onboard-node.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Check that root can reach the node over SSH. This must print `ok` without asking
|
||||||
|
for a password:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo ssh ubuntu@10.0.0.21 'sudo true && echo ok'
|
||||||
|
```
|
||||||
|
|
||||||
|
Then add the node. `--node` is its name in the cluster; use its hostname.
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo ./onboard-node.sh --node gpu01 --addr 10.0.0.21 \
|
||||||
|
--server https://10.0.0.10:7443 --ssh-user ubuntu
|
||||||
|
```
|
||||||
|
|
||||||
|
Replace `10.0.0.21` with the node's address, `10.0.0.10` with the control
|
||||||
|
plane's, and `ubuntu` with the user you SSH in as.
|
||||||
|
|
||||||
|
If the node has InfiniBand, name the fabric and its adapters, and give every
|
||||||
|
node on the same fabric the same name:
|
||||||
|
|
||||||
|
```
|
||||||
|
--fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
|
||||||
|
```
|
||||||
|
|
||||||
|
**Check:**
|
||||||
|
|
||||||
|
```
|
||||||
|
llmgw ctl nodes
|
||||||
|
```
|
||||||
|
|
||||||
|
The node appears within seconds. It then installs its engines by itself:
|
||||||
|
llama.cpp in about a minute, vLLM in a few more (about 8 GB). The cluster page
|
||||||
|
(`/admin`) shows the progress under each node. If a node cannot run vLLM, it
|
||||||
|
says why on the same page.
|
||||||
|
|
||||||
|
Repeat for each node.
|
||||||
|
|
||||||
|
## 4. Add a model
|
||||||
|
|
||||||
|
Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
|
||||||
|
Hugging Face, showing only what your nodes can run.
|
||||||
|
|
||||||
|
1. Click a model to read its card.
|
||||||
|
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
||||||
|
**Fetch**. It is downloaded into the mirror once, for every node.
|
||||||
|
3. When the download finishes, click **serve** on its row in the table above.
|
||||||
|
4. In the form, choose how it runs: **always on**, or **on demand**, which
|
||||||
|
loads it on the first request and stops it after 15 minutes unused. Click
|
||||||
|
**Add to the catalog**.
|
||||||
|
|
||||||
|
The model is placed on a node that fits it, and its status on `/admin` turns
|
||||||
|
`ready`.
|
||||||
|
|
||||||
|
**Check:** the model appears in the list on `/chat`. Ask it something.
|
||||||
|
|
||||||
|
## 5. Give people access
|
||||||
|
|
||||||
|
**A key for a program:** each person makes their own on the account page
|
||||||
|
(`/profile`). The key is shown once.
|
||||||
|
|
||||||
|
```
|
||||||
|
curl http://<control-plane>:8080/v1/chat/completions \
|
||||||
|
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Any OpenAI client works. Only the base URL changes:
|
||||||
|
`http://<control-plane>:8080/v1`.
|
||||||
|
|
||||||
|
**An editor:** the account page connects Continue, Codex or Cline. Pick the
|
||||||
|
tool and a model, and it gives a one-line command that writes the
|
||||||
|
configuration.
|
||||||
|
|
||||||
|
**Other people's accounts:** they sign up at `/signup`, and you approve them on
|
||||||
|
the people page (`/admin/users`).
|
||||||
|
|
||||||
|
## 6. Put it behind HTTPS
|
||||||
|
|
||||||
|
Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
|
||||||
|
reverse proxy in front with a real certificate. With Caddy, which gets the
|
||||||
|
certificate by itself:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo apt-get install -y caddy # Debian and Ubuntu
|
||||||
|
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
|
||||||
|
sudo systemctl reload caddy
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
|
||||||
|
through the proxy. There is an nginx configuration too:
|
||||||
|
[deploy/proxy/](../deploy/proxy/).
|
||||||
|
|
||||||
|
## When something does not work
|
||||||
|
|
||||||
|
| What you see | What to do |
|
||||||
|
|---|---|
|
||||||
|
| A node does not appear in `llmgw ctl nodes` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443, or the control plane's address is not in its certificate: run `server init` again with `--host`. |
|
||||||
|
| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
|
||||||
|
| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
|
||||||
|
| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
|
||||||
|
| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |
|
||||||
|
|
||||||
|
## Next
|
||||||
|
|
||||||
|
- [install.md](install.md): the same steps with every option explained.
|
||||||
|
- [fabrics.md](fabrics.md): InfiniBand and NVLink.
|
||||||
|
- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.
|
||||||
+3
-3
@@ -1,8 +1,8 @@
|
|||||||
# Quickstart: one machine, one model, ten minutes
|
# Quickstart: one machine, one model, ten minutes
|
||||||
|
|
||||||
Written for someone who wants to see llmgw answer a request on a single
|
Written for a developer who wants to see llmgw answer a request on a single
|
||||||
workstation before reading anything else. No systemd, no `sudo`. For a real
|
workstation, built from source, with no systemd and no `sudo`. **To install
|
||||||
cluster, see [install.md](install.md).
|
llmgw on servers, read [quick-install.md](quick-install.md) instead.**
|
||||||
|
|
||||||
What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
|
What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
|
||||||
of a real model served by llama.cpp.
|
of a real model served by llama.cpp.
|
||||||
|
|||||||
@@ -203,7 +203,7 @@ footer a{color:var(--muted)}
|
|||||||
fabric, and a model split across machines never straddles two.</p>
|
fabric, and a model split across machines never straddles two.</p>
|
||||||
<div class="cta">
|
<div class="cta">
|
||||||
<a class="btn" href="#download">Download{{with .Release}} {{.}}{{end}}</a>
|
<a class="btn" href="#download">Download{{with .Release}} {{.}}{{end}}</a>
|
||||||
<a class="btn ghost" href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/install.md">Install guide</a>
|
<a class="btn ghost" href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/quick-install.md">Install guide</a>
|
||||||
</div>
|
</div>
|
||||||
<div class="oneline" aria-label="Set up a control plane">
|
<div class="oneline" aria-label="Set up a control plane">
|
||||||
<code><span class="p">$ </span>sudo llmgw server init --admin you@example.com</code>
|
<code><span class="p">$ </span>sudo llmgw server init --admin you@example.com</code>
|
||||||
|
|||||||
Reference in new issue
Block a user