In a terminal and without --admin, server init shows a short form: the administrator's email, the other names nodes will use (with the addresses it found), SQLite or PostgreSQL, and the mirror, node and starter model. Its own lines then fill in as a list — read from the writer init already uses, so nothing in init changed — with downloads updating in place and the password in a box until the end. The whole run, password included, is printed again when the screen closes, so it is in the terminal's history. The last screen says how to add a node and to back up the authority's key. node add takes several addresses and adds them in turn, with a summary. Suggested fabrics carry what the adapters report them to be, so a name given for one node is used for the next on the same fabric without asking. Tried on empty VMs: init from the form through to the starter model serving, and two nodes added to that cluster in one command. The first attempt at the form lost its keystrokes to the terminal colour query in a pseudo-terminal that never answers it; a real terminal does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
190 lines
7.4 KiB
Markdown
190 lines
7.4 KiB
Markdown
# Quick install
|
|
|
|
From empty servers to a model answering requests, in six steps. This page
|
|
has only the commands and what to check after each; [install.md](install.md)
|
|
explains every choice.
|
|
|
|
## What you need
|
|
|
|
| | |
|
|
|---|---|
|
|
| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
|
|
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
|
|
| **root** | on the control plane, and `sudo` without a password on every node. |
|
|
| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. |
|
|
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
|
|
|
|
You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
|
|
themselves once they join.
|
|
|
|
## 1. Put llmgw on the control plane
|
|
|
|
```
|
|
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
|
|
| grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
|
|
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
|
|
curl -fsSLO $U/llmgw-$VERSION-linux-amd64 # -linux-arm64 on ARM
|
|
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
|
|
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
|
|
llmgw --version
|
|
```
|
|
|
|
`sha256sum` must print `OK`. If it does not, the download is damaged; do not
|
|
install it.
|
|
|
|
## 2. Set up the control plane
|
|
|
|
```
|
|
sudo llmgw server init
|
|
```
|
|
|
|
It asks for your email and a few choices, then sets up everything on this
|
|
machine, in a few minutes. In a script, give them as flags instead:
|
|
`sudo llmgw server init --admin you@example.com`.
|
|
|
|
- the service, its certificates and its configuration;
|
|
- your administrator account. **The password is printed once. Copy it now.**
|
|
- a model mirror, which downloads model files once for the whole cluster;
|
|
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
|
|
|
|
If nodes will reach this machine by a name or address it does not know itself
|
|
(a DNS name, a NAT address), add `--host` for each:
|
|
|
|
```
|
|
sudo llmgw server init --admin you@example.com --host llm.example.com
|
|
```
|
|
|
|
It is safe to run again. It keeps what exists and finishes what is missing.
|
|
|
|
**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
|
|
password, and ask something. The small model answers. Change the password on
|
|
the account page.
|
|
|
|
## 3. Add the GPU nodes
|
|
|
|
On the control plane:
|
|
|
|
```
|
|
sudo llmgw node add ubuntu@10.0.0.21
|
|
```
|
|
|
|
Several at once: `sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23`.
|
|
|
|
Replace `ubuntu` with the user you log in to the node as, and `10.0.0.21` with
|
|
its address. That user needs `sudo` without a password on the node.
|
|
|
|
It logs in over SSH and, while you watch:
|
|
|
|
1. installs llmgw on the node, the release for its processor, checksum checked;
|
|
2. runs `llmgw doctor` there, and stops if the node has a problem. Each problem
|
|
comes with the command that fixes it. Fix it and run the same command again;
|
|
3. gives the node its certificate and configuration, and starts its agent;
|
|
4. waits for the node to join and install llama.cpp.
|
|
|
|
The node's name in the cluster is its hostname. vLLM installs by itself after
|
|
that, in a few more minutes on a GPU node (about 8 GB). The cluster page
|
|
(`/admin`) shows its progress, or why the node cannot run it.
|
|
|
|
**InfiniBand.** When the node's adapters are up, `node add` shows them grouped
|
|
by the fabric they are on and asks for each fabric's name. Give every node on
|
|
the same fabric the same name. In a script, give it instead:
|
|
|
|
```
|
|
sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
|
|
```
|
|
|
|
Once two nodes on a fabric are in, check they reach each other at the link's
|
|
rate before placing anything across them:
|
|
|
|
```
|
|
llmgw ctl fabric-test gpu01 gpu02
|
|
```
|
|
|
|
**If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
|
|
own keys instead, log in to the control plane with `ssh -A` and keep your
|
|
agent through sudo:
|
|
|
|
```
|
|
sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21
|
|
```
|
|
|
|
**Check:** `llmgw ctl nodes` lists the node. Running `node add` again on the
|
|
same node is safe: it renews its certificate and configuration, and keeps what
|
|
runs there.
|
|
|
|
A machine retired for good is removed with `llmgw node remove <name>` once its
|
|
agent is stopped.
|
|
|
|
## 4. Add a model
|
|
|
|
Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
|
|
Hugging Face, showing only what your nodes can run.
|
|
|
|
1. Click a model to read its card.
|
|
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
|
**Fetch**. It is downloaded into the mirror once, for every node.
|
|
3. When the download finishes, click **serve** on its row in the table above.
|
|
4. In the form, choose how it runs: **always on**, or **on demand**, which
|
|
loads it on the first request and stops it after 15 minutes unused. Click
|
|
**Add to the catalog**.
|
|
|
|
The model is placed on a node that fits it, and its status on `/admin` turns
|
|
`ready`.
|
|
|
|
**Check:** the model appears in the list on `/chat`. Ask it something.
|
|
|
|
## 5. Give people access
|
|
|
|
**A key for a program:** each person makes their own on the account page
|
|
(`/profile`). The key is shown once.
|
|
|
|
```
|
|
curl http://<control-plane>:8080/v1/chat/completions \
|
|
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
|
|
```
|
|
|
|
Any OpenAI client works. Only the base URL changes:
|
|
`http://<control-plane>:8080/v1`.
|
|
|
|
**An editor:** the account page connects Continue, Codex or Cline. Pick the
|
|
tool and a model, and it gives a one-line command that writes the
|
|
configuration.
|
|
|
|
**Other people's accounts:** they sign up at `/signup`, and you approve them on
|
|
the people page (`/admin/users`).
|
|
|
|
## 6. Put it behind HTTPS
|
|
|
|
Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
|
|
reverse proxy in front with a real certificate. With Caddy, which gets the
|
|
certificate by itself:
|
|
|
|
```
|
|
sudo apt-get install -y caddy # Debian and Ubuntu
|
|
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
|
|
sudo systemctl reload caddy
|
|
```
|
|
|
|
Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
|
|
through the proxy. There is an nginx configuration too:
|
|
[deploy/proxy/](../deploy/proxy/).
|
|
|
|
## When something does not work
|
|
|
|
| What you see | What to do |
|
|
|---|---|
|
|
| `node add` stops at `join` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443 on the control plane. |
|
|
| `node add` says the address is not in the certificate | Run `sudo llmgw server init --host <that address>`, then `node add` again. |
|
|
| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
|
|
| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
|
|
| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
|
|
| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |
|
|
|
|
## Next
|
|
|
|
- [install.md](install.md): the same steps with every option explained.
|
|
- [fabrics.md](fabrics.md): InfiniBand and NVLink.
|
|
- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.
|