llmgw/docs/quick-install.md

Quick install

From empty servers to a model answering requests, in six steps. This page has only the commands and what to check after each; install.md explains every choice.

What you need

One control plane machine	Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk.
GPU nodes (optional to start)	The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100).
root	on the control plane, and sudo without a password on every node.
SSH	from root on the control plane to each node.
Network	nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models.

You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that themselves once they join.

1. Put llmgw on the control plane

VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64          # -linux-arm64 on ARM
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version

sha256sum must print OK. If it does not, the download is damaged; do not install it.

2. Set up the control plane

sudo llmgw server init --admin you@example.com

This one command sets up everything on this machine, and takes a few minutes:

• the service, its certificates and its configuration;
• your administrator account. The password is printed once. Copy it now.
• a model mirror, which downloads model files once for the whole cluster;
• this machine as a first node, and a small model (Qwen2.5 0.5B) on it.

If nodes will reach this machine by a name or address it does not know itself (a DNS name, a NAT address), add --host for each:

sudo llmgw server init --admin you@example.com --host llm.example.com

It is safe to run again. It keeps what exists and finishes what is missing.

Check: open http://<control-plane>:8080/chat, sign in with the printed password, and ask something. The small model answers. Change the password on the account page.

3. Add the GPU nodes

On each node, check it first. Set VERSION and U as in step 1, then:

curl -fsSLO $U/llmgw-$VERSION-linux-amd64
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
sudo llmgw doctor

Fix every FAIL line before going on. The fix is printed under each line.

On the control plane, get the onboarding script once:

curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/raw/tag/$VERSION/deploy/bootstrap/onboard-node.sh
chmod +x onboard-node.sh

Check that root can reach the node over SSH. This must print ok without asking for a password:

sudo ssh ubuntu@10.0.0.21 'sudo true && echo ok'

Then add the node. --node is its name in the cluster; use its hostname.

sudo ./onboard-node.sh --node gpu01 --addr 10.0.0.21 \
  --server https://10.0.0.10:7443 --ssh-user ubuntu

Replace 10.0.0.21 with the node's address, 10.0.0.10 with the control plane's, and ubuntu with the user you SSH in as.

If the node has InfiniBand, name the fabric and its adapters, and give every node on the same fabric the same name:

  --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3

Check:

llmgw ctl nodes

The node appears within seconds. It then installs its engines by itself: llama.cpp in about a minute, vLLM in a few more (about 8 GB). The cluster page (/admin) shows the progress under each node. If a node cannot run vLLM, it says why on the same page.

Repeat for each node.

4. Add a model

Open /admin/mirror and choose Explore models. It lists what is popular on Hugging Face, showing only what your nodes can run.

1. Click a model to read its card.
2. Pick one of its files, usually a q4_k_m for a first try, and click Fetch. It is downloaded into the mirror once, for every node.
3. When the download finishes, click serve on its row in the table above.
4. In the form, choose how it runs: always on, or on demand, which loads it on the first request and stops it after 15 minutes unused. Click Add to the catalog.

The model is placed on a node that fits it, and its status on /admin turns ready.

Check: the model appears in the list on /chat. Ask it something.

5. Give people access

A key for a program: each person makes their own on the account page (/profile). The key is shown once.

curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'

Any OpenAI client works. Only the base URL changes: http://<control-plane>:8080/v1.

An editor: the account page connects Continue, Codex or Cline. Pick the tool and a model, and it gives a one-line command that writes the configuration.

Other people's accounts: they sign up at /signup, and you approve them on the people page (/admin/users).

6. Put it behind HTTPS

Port 8080 is plain HTTP. Before users outside a trusted network use it, put a reverse proxy in front with a real certificate. With Caddy, which gets the certificate by itself:

sudo apt-get install -y caddy          # Debian and Ubuntu
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
sudo systemctl reload caddy

Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go through the proxy. There is an nginx configuration too: deploy/proxy/.

When something does not work

What you see	What to do
A node does not appear in llmgw ctl nodes	sudo journalctl -u llmgw-agent -n 50 on the node. Usually it cannot reach port 7443, or the control plane's address is not in its certificate: run server init again with --host.
A node says it cannot run vLLM	The reason is on /admin: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free.
A model shows not placed	llmgw ctl explain-placement <model> names each node and why it was not used.
A reply stops after 60 seconds behind nginx	Use the nginx configuration from the release. Its timeouts are set for slow models.
Anything else about a node	sudo llmgw doctor on it. --json writes everything it read, to send to whoever helps you.

Next

• install.md: the same steps with every option explained.
• fabrics.md: InfiniBand and NVLink.
• onboarding.md: accounts, projects, budgets and keys.