Ilker ManapandClaude Opus 5.5
a1b5daceb8
In a terminal and without --admin, server init shows a short form: the administrator's email, the other names nodes will use (with the addresses it found), SQLite or PostgreSQL, and the mirror, node and starter model. Its own lines then fill in as a list — read from the writer init already uses, so nothing in init changed — with downloads updating in place and the password in a box until the end. The whole run, password included, is printed again when the screen closes, so it is in the terminal's history. The last screen says how to add a node and to back up the authority's key. node add takes several addresses and adds them in turn, with a summary. Suggested fabrics carry what the adapters report them to be, so a name given for one node is used for the next on the same fabric without asking. Tried on empty VMs: init from the form through to the starter model serving, and two nodes added to that cluster in one command. The first attempt at the form lost its keystrokes to the terminal colour query in a pseudo-terminal that never answers it; a real terminal does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7.4 KiB
Quick install
From empty servers to a model answering requests, in six steps. This page has only the commands and what to check after each; install.md explains every choice.
What you need
| One control plane machine | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
| GPU nodes (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
| root | on the control plane, and sudo without a password on every node. |
| SSH | from the control plane to each node, as a user with passwordless sudo there. |
| Network | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that themselves once they join.
1. Put llmgw on the control plane
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
| grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64 # -linux-arm64 on ARM
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version
sha256sum must print OK. If it does not, the download is damaged; do not
install it.
2. Set up the control plane
sudo llmgw server init
It asks for your email and a few choices, then sets up everything on this
machine, in a few minutes. In a script, give them as flags instead:
sudo llmgw server init --admin you@example.com.
- the service, its certificates and its configuration;
- your administrator account. The password is printed once. Copy it now.
- a model mirror, which downloads model files once for the whole cluster;
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
If nodes will reach this machine by a name or address it does not know itself
(a DNS name, a NAT address), add --host for each:
sudo llmgw server init --admin you@example.com --host llm.example.com
It is safe to run again. It keeps what exists and finishes what is missing.
Check: open http://<control-plane>:8080/chat, sign in with the printed
password, and ask something. The small model answers. Change the password on
the account page.
3. Add the GPU nodes
On the control plane:
sudo llmgw node add ubuntu@10.0.0.21
Several at once: sudo llmgw node add ubuntu@10.0.0.21 ubuntu@10.0.0.22 ubuntu@10.0.0.23.
Replace ubuntu with the user you log in to the node as, and 10.0.0.21 with
its address. That user needs sudo without a password on the node.
It logs in over SSH and, while you watch:
- installs llmgw on the node, the release for its processor, checksum checked;
- runs
llmgw doctorthere, and stops if the node has a problem. Each problem comes with the command that fixes it. Fix it and run the same command again; - gives the node its certificate and configuration, and starts its agent;
- waits for the node to join and install llama.cpp.
The node's name in the cluster is its hostname. vLLM installs by itself after
that, in a few more minutes on a GPU node (about 8 GB). The cluster page
(/admin) shows its progress, or why the node cannot run it.
InfiniBand. When the node's adapters are up, node add shows them grouped
by the fabric they are on and asks for each fabric's name. Give every node on
the same fabric the same name. In a script, give it instead:
sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
Once two nodes on a fabric are in, check they reach each other at the link's rate before placing anything across them:
llmgw ctl fabric-test gpu01 gpu02
If it cannot log in. sudo runs SSH as root, with root's keys. To use your
own keys instead, log in to the control plane with ssh -A and keep your
agent through sudo:
sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21
Check: llmgw ctl nodes lists the node. Running node add again on the
same node is safe: it renews its certificate and configuration, and keeps what
runs there.
A machine retired for good is removed with llmgw node remove <name> once its
agent is stopped.
4. Add a model
Open /admin/mirror and choose Explore models. It lists what is popular on
Hugging Face, showing only what your nodes can run.
- Click a model to read its card.
- Pick one of its files, usually a
q4_k_mfor a first try, and click Fetch. It is downloaded into the mirror once, for every node. - When the download finishes, click serve on its row in the table above.
- In the form, choose how it runs: always on, or on demand, which loads it on the first request and stops it after 15 minutes unused. Click Add to the catalog.
The model is placed on a node that fits it, and its status on /admin turns
ready.
Check: the model appears in the list on /chat. Ask it something.
5. Give people access
A key for a program: each person makes their own on the account page
(/profile). The key is shown once.
curl http://<control-plane>:8080/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
Any OpenAI client works. Only the base URL changes:
http://<control-plane>:8080/v1.
An editor: the account page connects Continue, Codex or Cline. Pick the tool and a model, and it gives a one-line command that writes the configuration.
Other people's accounts: they sign up at /signup, and you approve them on
the people page (/admin/users).
6. Put it behind HTTPS
Port 8080 is plain HTTP. Before users outside a trusted network use it, put a reverse proxy in front with a real certificate. With Caddy, which gets the certificate by itself:
sudo apt-get install -y caddy # Debian and Ubuntu
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
sudo systemctl reload caddy
Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go through the proxy. There is an nginx configuration too: deploy/proxy/.
When something does not work
| What you see | What to do |
|---|---|
node add stops at join |
sudo journalctl -u llmgw-agent -n 50 on the node. Usually it cannot reach port 7443 on the control plane. |
node add says the address is not in the certificate |
Run sudo llmgw server init --host <that address>, then node add again. |
| A node says it cannot run vLLM | The reason is on /admin: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
A model shows not placed |
llmgw ctl explain-placement <model> names each node and why it was not used. |
| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
| Anything else about a node | sudo llmgw doctor on it. --json writes everything it read, to send to whoever helps you. |
Next
- install.md: the same steps with every option explained.
- fabrics.md: InfiniBand and NVLink.
- onboarding.md: accounts, projects, budgets and keys.