llmgw/docs/quick-install.md

# Quick install

From empty servers to a model answering requests, in six steps. This page
has only the commands and what to check after each; [install.md](install.md)
explains every choice.

## What you need

| | |
|---|---|
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 12 GB RAM when it is also a node running the starter model and the two judges, which setup checks before it installs anything; four cores or more, or the judges on machines of their own, keep a judgment within its wait. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |

You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
themselves once they join.

## 1. Put llmgw on the control plane

```
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64          # -linux-arm64 on ARM
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version
```

`sha256sum` must print `OK`. If it does not, the download is damaged; do not
install it.

## 2. Set up the control plane

```
sudo llmgw
```

With nothing installed, `llmgw` opens the setup screens. They ask whether
this is **one machine or three** (high availability), and for three, the
other two: address, user, and an ssh key, the ssh agent or a password. They
check every machine before installing anything — system, disk, clock, ports,
internet, whether the machines reach one another — show the plan, and
install:

- the service, its certificates and its configuration, and a firewall for
  its internal ports;
- with three machines, the store replicated across them (PostgreSQL under
  Patroni);
- your administrator account. **The password is shown once. Copy it now.**
- a model mirror, which downloads model files once for the whole cluster;
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
- guardrails on every request, with a judge model (Qwen3Guard-Gen 0.6B, two
  replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how
  to loosen it: [install.md](install.md#2-the-control-plane).

Run again later, `sudo llmgw` is the console: the cluster's state and
everything you can do to it ([console.md](console.md)). Every step below is
also there, under **Do**.

**In a script**, the same for one machine:

```
sudo llmgw server init --admin you@example.com
```

If nodes will reach this machine by a name or address it does not know itself
(a DNS name, a NAT address), add `--host` for each:
`--host llm.example.com`. It is safe to run again: it keeps what exists and
finishes what is missing.

**The mirror on its own machine**, when it wants a larger disk than the
control plane has: first on that machine, with llmgw installed as in step 1,

```
sudo llmgw mirror init --control-plane <control plane address>
```

then on the control plane `sudo llmgw server init --mirror http://<mirror
address>:8082`. The mirror serves model files to every node, and takes
orders only from the control plane. Untick "This machine as a node" in the
form (or add `--no-node`) to keep models off the control plane; the starter
model then starts on the first node you add.

**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
password, and ask something. The small model answers. Change the password on
the account page.

## 3. Add the GPU nodes

In the console, **Do → Add a node**. Or on the control plane:

```
sudo llmgw node add ubuntu@10.0.0.21
```

Several at once: `sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23`.

Replace `ubuntu` with the user you log in to the node as, and `10.0.0.21` with
its address. That user needs `sudo` without a password on the node.

It logs in over SSH and, while you watch:

1. installs llmgw on the node, the release for its processor, checksum checked;
2. runs `llmgw doctor` there, and stops if the node has a problem. Each problem
   comes with the command that fixes it. Fix it and run the same command again;
3. gives the node its certificate and configuration, and starts its agent;
4. waits for the node to join and install llama.cpp.

The node's name in the cluster is its hostname. vLLM installs by itself after
that, in a few more minutes on a GPU node (about 8 GB). The cluster page
(`/admin`) shows its progress, or why the node cannot run it.

**InfiniBand.** When the node's adapters are up, `node add` shows them grouped
by the fabric they are on and asks for each fabric's name. Give every node on
the same fabric the same name. In a script, give it instead:

```
sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
```

Once two nodes on a fabric are in, check they reach each other at the link's
rate before placing anything across them:

```
llmgw ctl fabric-test gpu01 gpu02
```

**If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
own keys instead, log in to the control plane with `ssh -A` and keep your
agent through sudo:

```
sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21
```

**Check:** `llmgw ctl nodes` lists the node. Running `node add` again on the
same node is safe: it renews its certificate and configuration, and keeps what
runs there.

A machine retired for good is removed with `llmgw node remove <name>` once its
agent is stopped.

## 4. Add a model

Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
Hugging Face, showing only what your nodes can run: judged by the cluster as it
is now — the largest card and its compute capability, the most memory on one
node — so a model too big for them, in a form those cards lack (FP8 needs 8.9),
or in neither GGUF nor safetensors is left out, and the page says how many.
Without GPUs it lists GGUF only. Untick **only what these nodes can run** to
see everything. On a model's page a file too big for the cluster is shown
without a **Fetch** button, with what it would need.

The **CPU tok/s** column is about how fast one reply would come on the CPU of
the fastest node whose memory holds the model at Q4_K_M; a model's page gives
it for each GGUF file by its own size. Every node's agent measures how fast its
memory reads when it starts (it is in the agent's log as `memory_gbs`), and a
token on a CPU reads every weight once: the estimate is that speed over the
model's size — for a model of experts named like `30B-A3B`, its active part.
On the test cluster it came within 15% of what llama.cpp measured. Several
replies at once share the reads and each is slower.

**GPU tok/s** is the same on the cards, for the layout this cluster would
give the model — the fewest cards that hold it, of each class, the fastest of
those; its tooltip says which ("2× h100-80g, NVLink, vLLM", "2 nodes × 1×
a100-80g over ib0 (InfiniBand 200 Gb/s), vLLM"). A card's memory bandwidth is
known from its model name (an H100 SXM and an H100 PCIe differ by half); a card
not in the table gets no figure. Tensor parallel across a node's cards divides
what each reads, and adds two all-reduces a layer, short over NVLink and
longer over PCIe; llama.cpp across cards splits by layer, so the cards take
turns and a reply is no faster than on one; a pipeline across nodes takes turns
too, and adds a hop over the fabric for each node. One card's figures were
measured on the test cluster's RTX 6000; tensor parallel and the fabric are
from what NCCL is known to take, not measured here.

1. Click a model to read its card.
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
   **Fetch**. It is downloaded into the mirror once, for every node.
3. When the download finishes, click **serve** on its row in the table above.
4. In the form, choose how it runs: **always on**, or **on demand**, which
   loads it on the first request and stops it after 15 minutes unused. Click
   **Add to the catalog**.

The model is placed on a node that fits it, and its status on `/admin` turns
`ready`.

**Check:** the model appears in the list on `/chat`. Ask it something.

## 5. Give people access

**A key for a program:** each person makes their own on the account page
(`/profile`). The key is shown once.

```
curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
```

Any OpenAI client works. Only the base URL changes:
`http://<control-plane>:8080/v1`.

**An editor:** the account page connects Continue, Codex or Cline. Pick the
tool and a model, and it gives a one-line command that writes the
configuration.

**Other people's accounts:** they sign up at `/signup`, and you approve them on
the people page (`/admin/users`).

## 6. Put it behind HTTPS

Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
reverse proxy in front with a real certificate. With Caddy, which gets the
certificate by itself:

```
sudo apt-get install -y caddy          # Debian and Ubuntu
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
sudo systemctl reload caddy
```

Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
through the proxy. There is an nginx configuration too:
[deploy/proxy/](../deploy/proxy/).

## When something does not work

| What you see | What to do |
|---|---|
| `node add` stops at `join` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443 on the control plane. |
| `node add` says the address is not in the certificate | Run `sudo llmgw server init --host <that address>`, then `node add` again. |
| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |

## Next

- [install.md](install.md): the same steps with every option explained, and
  running the cluster after: monitoring, notifications, single sign-on,
  backups, upgrades.
- [console.md](console.md): everything the console does.
- [fabrics.md](fabrics.md): InfiniBand and NVLink.
- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.