llmgw/docs/quick-install.md

# Quick install

From empty servers to a model answering requests, in six steps. This page
has only the commands and what to check after each; [install.md](install.md)
explains every choice.

## What you need

| | |
|---|---|
| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
| **root** | on the control plane, and `sudo` without a password on every node. |
| **SSH** | from root on the control plane to each node. |
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |

You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
themselves once they join.

## 1. Put llmgw on the control plane

```
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64          # -linux-arm64 on ARM
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version
```

`sha256sum` must print `OK`. If it does not, the download is damaged; do not
install it.

## 2. Set up the control plane

```
sudo llmgw server init --admin you@example.com
```

This one command sets up everything on this machine, and takes a few minutes:

- the service, its certificates and its configuration;
- your administrator account. **The password is printed once. Copy it now.**
- a model mirror, which downloads model files once for the whole cluster;
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.

If nodes will reach this machine by a name or address it does not know itself
(a DNS name, a NAT address), add `--host` for each:

```
sudo llmgw server init --admin you@example.com --host llm.example.com
```

It is safe to run again. It keeps what exists and finishes what is missing.

**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
password, and ask something. The small model answers. Change the password on
the account page.

## 3. Add the GPU nodes

**On each node**, check it first. Set `VERSION` and `U` as in step 1, then:

```
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
sudo llmgw doctor
```

Fix every `FAIL` line before going on. The fix is printed under each line.

**On the control plane**, get the onboarding script once:

```
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/raw/tag/$VERSION/deploy/bootstrap/onboard-node.sh
chmod +x onboard-node.sh
```

Check that root can reach the node over SSH. This must print `ok` without asking
for a password:

```
sudo ssh ubuntu@10.0.0.21 'sudo true && echo ok'
```

Then add the node. `--node` is its name in the cluster; use its hostname.

```
sudo ./onboard-node.sh --node gpu01 --addr 10.0.0.21 \
  --server https://10.0.0.10:7443 --ssh-user ubuntu
```

Replace `10.0.0.21` with the node's address, `10.0.0.10` with the control
plane's, and `ubuntu` with the user you SSH in as.

If the node has InfiniBand, name the fabric and its adapters, and give every
node on the same fabric the same name:

```
  --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
```

**Check:**

```
llmgw ctl nodes
```

The node appears within seconds. It then installs its engines by itself:
llama.cpp in about a minute, vLLM in a few more (about 8 GB). The cluster page
(`/admin`) shows the progress under each node. If a node cannot run vLLM, it
says why on the same page.

Repeat for each node.

## 4. Add a model

Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
Hugging Face, showing only what your nodes can run.

1. Click a model to read its card.
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
   **Fetch**. It is downloaded into the mirror once, for every node.
3. When the download finishes, click **serve** on its row in the table above.
4. In the form, choose how it runs: **always on**, or **on demand**, which
   loads it on the first request and stops it after 15 minutes unused. Click
   **Add to the catalog**.

The model is placed on a node that fits it, and its status on `/admin` turns
`ready`.

**Check:** the model appears in the list on `/chat`. Ask it something.

## 5. Give people access

**A key for a program:** each person makes their own on the account page
(`/profile`). The key is shown once.

```
curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
```

Any OpenAI client works. Only the base URL changes:
`http://<control-plane>:8080/v1`.

**An editor:** the account page connects Continue, Codex or Cline. Pick the
tool and a model, and it gives a one-line command that writes the
configuration.

**Other people's accounts:** they sign up at `/signup`, and you approve them on
the people page (`/admin/users`).

## 6. Put it behind HTTPS

Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
reverse proxy in front with a real certificate. With Caddy, which gets the
certificate by itself:

```
sudo apt-get install -y caddy          # Debian and Ubuntu
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
sudo systemctl reload caddy
```

Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
through the proxy. There is an nginx configuration too:
[deploy/proxy/](../deploy/proxy/).

## When something does not work

| What you see | What to do |
|---|---|
| A node does not appear in `llmgw ctl nodes` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443, or the control plane's address is not in its certificate: run `server init` again with `--host`. |
| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |

## Next

- [install.md](install.md): the same steps with every option explained.
- [fabrics.md](fabrics.md): InfiniBand and NVLink.
- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.