llmgw/docs/quick-install.md

Quick install

From empty servers to a model answering requests, in six steps. This page has only the commands and what to check after each; install.md explains every choice.

What you need

A control plane	One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 12 GB RAM when it is also a node running the starter model and the two judges, which setup checks before it installs anything; four cores or more, or the judges on machines of their own, keep a judgment within its wait.
GPU nodes (optional to start)	The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM.
root	on the control plane. On the other two of a three-machine setup, sudo (its password is asked for if it needs one); on nodes, sudo without a password.
SSH	from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; node add uses ssh keys. A machine nothing may log into joins with a code instead (llmgw join).
Network	nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models.

You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that themselves once they join.

1. Put llmgw on the control plane

VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64          # -linux-arm64 on ARM
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version

sha256sum must print OK. If it does not, the download is damaged; do not install it.

2. Set up the control plane

sudo llmgw

With nothing installed, llmgw opens the setup screens. They ask whether this is one machine or three (high availability), and for three, the other two: address, user, and an ssh key, the ssh agent or a password. They check every machine before installing anything — system, disk, clock, ports, internet, whether the machines reach one another — show the plan, and install:

• the service, its certificates and its configuration, and a firewall for its internal ports;
• with three machines, the store replicated across them (PostgreSQL under Patroni);
• your administrator account. The password is shown once. Copy it now.
• a model mirror, which downloads model files once for the whole cluster;
• this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
• guardrails on every request, with a judge model (Qwen3Guard-Gen 0.6B, two replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how to loosen it: install.md.

Run again later, sudo llmgw is the console: the cluster's state and everything you can do to it (console.md). Every step below is also there, under Do.

In a script, the same for one machine:

sudo llmgw server init --admin you@example.com

If nodes will reach this machine by a name or address it does not know itself (a DNS name, a NAT address), add --host for each: --host llm.example.com. It is safe to run again: it keeps what exists and finishes what is missing.

The mirror on its own machine, when it wants a larger disk than the control plane has: first on that machine, with llmgw installed as in step 1,

sudo llmgw mirror init --control-plane <control plane address>

then on the control plane sudo llmgw server init --mirror http://<mirror address>:8082. The mirror serves model files to every node, and takes orders only from the control plane. Untick "This machine as a node" in the form (or add --no-node) to keep models off the control plane; the starter model then starts on the first node you add.

Check: open http://<control-plane>:8080/chat, sign in with the printed password, and ask something. The small model answers. Change the password on the account page.

3. Add the GPU nodes

In the console, Do → Add a node. Or on the control plane:

sudo llmgw node add ubuntu@10.0.0.21

Several at once: sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23.

Replace ubuntu with the user you log in to the node as, and 10.0.0.21 with its address. That user needs sudo without a password on the node.

It logs in over SSH and, while you watch:

1. installs llmgw on the node, the release for its processor, checksum checked;
2. runs llmgw doctor there, and stops if the node has a problem. Each problem comes with the command that fixes it. Fix it and run the same command again;
3. gives the node its certificate and configuration, and starts its agent;
4. waits for the node to join and install llama.cpp.

The node's name in the cluster is its hostname. vLLM installs by itself after that, in a few more minutes on a GPU node (about 8 GB). The cluster page (/admin) shows its progress, or why the node cannot run it.

InfiniBand. When the node's adapters are up, node add shows them grouped by the fabric they are on and asks for each fabric's name. Give every node on the same fabric the same name. In a script, give it instead:

sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3

Once two nodes on a fabric are in, check they reach each other at the link's rate before placing anything across them:

llmgw ctl fabric-test gpu01 gpu02

If it cannot log in. sudo runs SSH as root, with root's keys. To use your own keys instead, log in to the control plane with ssh -A and keep your agent through sudo:

sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21

Check: llmgw ctl nodes lists the node. Running node add again on the same node is safe: it renews its certificate and configuration, and keeps what runs there.

A machine retired for good is removed with llmgw node remove <name> once its agent is stopped.

4. Add a model

Open /admin/mirror and choose Explore models. It lists what is popular on Hugging Face, showing only what your nodes can run: judged by the cluster as it is now — the largest card and its compute capability, the most memory on one node — so a model too big for them, in a form those cards lack (FP8 needs 8.9), or in neither GGUF nor safetensors is left out, and the page says how many. Without GPUs it lists GGUF only. Untick only what these nodes can run to see everything. On a model's page a file too big for the cluster is shown without a Fetch button, with what it would need.

The CPU tok/s column is about how fast one reply would come on the CPU of the fastest node whose memory holds the model at Q4_K_M; a model's page gives it for each GGUF file by its own size. Every node's agent measures how fast its memory reads when it starts (it is in the agent's log as memory_gbs), and a token on a CPU reads every weight once: the estimate is that speed over the model's size — for a model of experts named like 30B-A3B, its active part. On the test cluster it came within 15% of what llama.cpp measured. Several replies at once share the reads and each is slower.

GPU tok/s is the same on the cards, for the layout this cluster would give the model — the fewest cards that hold it, of each class, the fastest of those; its tooltip says which ("2× h100-80g, NVLink, vLLM", "2 nodes × 1× a100-80g over ib0 (InfiniBand 200 Gb/s), vLLM"). A card's memory bandwidth is known from its model name (an H100 SXM and an H100 PCIe differ by half); a card not in the table gets no figure. Tensor parallel across a node's cards divides what each reads, and adds two all-reduces a layer, short over NVLink and longer over PCIe; llama.cpp across cards splits by layer, so the cards take turns and a reply is no faster than on one; a pipeline across nodes takes turns too, and adds a hop over the fabric for each node. One card's figures were measured on the test cluster's RTX 6000; tensor parallel and the fabric are from what NCCL is known to take, not measured here.

1. Click a model to read its card.
2. Pick one of its files, usually a q4_k_m for a first try, and click Fetch. It is downloaded into the mirror once, for every node.
3. When the download finishes, click serve on its row in the table above.
4. In the form, choose how it runs: always on, or on demand, which loads it on the first request and stops it after 15 minutes unused. Click Add to the catalog.

The model is placed on a node that fits it, and its status on /admin turns ready.

Check: the model appears in the list on /chat. Ask it something.

5. Give people access

A key for a program: each person makes their own on the account page (/profile). The key is shown once.

curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'

Any OpenAI client works. Only the base URL changes: http://<control-plane>:8080/v1.

An editor: the account page connects Continue, Codex or Cline. Pick the tool and a model, and it gives a one-line command that writes the configuration.

Other people's accounts: they sign up at /signup, and you approve them on the people page (/admin/users).

6. Put it behind HTTPS

Port 8080 is plain HTTP. Before users outside a trusted network use it, put a reverse proxy in front with a real certificate. With Caddy, which gets the certificate by itself:

sudo apt-get install -y caddy          # Debian and Ubuntu
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
sudo systemctl reload caddy

Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go through the proxy. There is an nginx configuration too: deploy/proxy/.

When something does not work

What you see	What to do
node add stops at join	sudo journalctl -u llmgw-agent -n 50 on the node. Usually it cannot reach port 7443 on the control plane.
node add says the address is not in the certificate	Run sudo llmgw server init --host <that address>, then node add again.
A node says it cannot run vLLM	The reason is on /admin: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free.
A model shows not placed	llmgw ctl explain-placement <model> names each node and why it was not used.
A reply stops after 60 seconds behind nginx	Use the nginx configuration from the release. Its timeouts are set for slow models.
Anything else about a node	sudo llmgw doctor on it. --json writes everything it read, to send to whoever helps you.

Next

• install.md: the same steps with every option explained, and running the cluster after: monitoring, notifications, single sign-on, backups, upgrades.
• console.md: everything the console does.
• fabrics.md: InfiniBand and NVLink.
• onboarding.md: accounts, projects, budgets and keys.