ilker/llmgw

0 Fork 1

Files

T

Ilker ManapandClaude Opus 5.5 7bb360fc78 Measure two nodes' fabric rail by rail: llmgw ctl fabric-test

doctor checks a node alone, and every check can pass while two nodes still
cannot reach each other. ctl fabric-test A B asks B's agent for an
ib_write_bw server and A's for the client, adapter by adapter as NCCL pairs
them, one rail at a time, and grades each against its link's rate. It goes
through the agents' streams, so there is no SSH between nodes; the agent
installs perftest the first time, and names the RoCE v2 GID itself.

The result is read from perftest's table by its heading, not a column
number. Agents now list the requests they answer beyond the usual
(Register.features), and the control plane refuses a test on one too old to
answer instead of waiting minutes for it. The forwarding dispatcher carries
the new capability, and its test says so: a missing forward would have made
the command quietly unavailable in production.

Not yet run over real InfiniBand or RoCE; this cluster has neither.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-09-29 21:29:23 +02:00

7.2 KiB

Raw Blame History

Quick install

From empty servers to a model answering requests, in six steps. This page has only the commands and what to check after each; install.md explains every choice.

What you need


One control plane machine	Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk.
GPU nodes (optional to start)	The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100).
root	on the control plane, and `sudo` without a password on every node.
SSH	from the control plane to each node, as a user with passwordless `sudo` there.
Network	nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models.

You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that themselves once they join.

1. Put llmgw on the control plane

VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64          # -linux-arm64 on ARM
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version

sha256sum must print OK. If it does not, the download is damaged; do not install it.

2. Set up the control plane

sudo llmgw server init --admin you@example.com

This one command sets up everything on this machine, and takes a few minutes:

the service, its certificates and its configuration;
your administrator account. The password is printed once. Copy it now.
a model mirror, which downloads model files once for the whole cluster;
this machine as a first node, and a small model (Qwen2.5 0.5B) on it.

If nodes will reach this machine by a name or address it does not know itself (a DNS name, a NAT address), add --host for each:

sudo llmgw server init --admin you@example.com --host llm.example.com

It is safe to run again. It keeps what exists and finishes what is missing.

Check: open http://<control-plane>:8080/chat, sign in with the printed password, and ask something. The small model answers. Change the password on the account page.

3. Add the GPU nodes

On the control plane, for each node:

sudo llmgw node add ubuntu@10.0.0.21

Replace ubuntu with the user you log in to the node as, and 10.0.0.21 with its address. That user needs sudo without a password on the node.

It logs in over SSH and, while you watch:

installs llmgw on the node, the release for its processor, checksum checked;
runs llmgw doctor there, and stops if the node has a problem. Each problem comes with the command that fixes it. Fix it and run the same command again;
gives the node its certificate and configuration, and starts its agent;
waits for the node to join and install llama.cpp.

The node's name in the cluster is its hostname. vLLM installs by itself after that, in a few more minutes on a GPU node (about 8 GB). The cluster page (/admin) shows its progress, or why the node cannot run it.

InfiniBand. When the node's adapters are up, node add shows them grouped by the fabric they are on and asks for each fabric's name. Give every node on the same fabric the same name. In a script, give it instead:

sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3

Once two nodes on a fabric are in, check they reach each other at the link's rate before placing anything across them:

llmgw ctl fabric-test gpu01 gpu02

If it cannot log in. sudo runs SSH as root, with root's keys. To use your own keys instead, log in to the control plane with ssh -A and keep your agent through sudo:

sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21

Check: llmgw ctl nodes lists the node. Running node add again on the same node is safe: it renews its certificate and configuration, and keeps what runs there.

A machine retired for good is removed with llmgw node remove <name> once its agent is stopped.

4. Add a model

Open /admin/mirror and choose Explore models. It lists what is popular on Hugging Face, showing only what your nodes can run.

Click a model to read its card.
Pick one of its files, usually a q4_k_m for a first try, and click Fetch. It is downloaded into the mirror once, for every node.
When the download finishes, click serve on its row in the table above.
In the form, choose how it runs: always on, or on demand, which loads it on the first request and stops it after 15 minutes unused. Click Add to the catalog.

The model is placed on a node that fits it, and its status on /admin turns ready.

Check: the model appears in the list on /chat. Ask it something.

5. Give people access

A key for a program: each person makes their own on the account page (/profile). The key is shown once.

curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'

Any OpenAI client works. Only the base URL changes: http://<control-plane>:8080/v1.

An editor: the account page connects Continue, Codex or Cline. Pick the tool and a model, and it gives a one-line command that writes the configuration.

Other people's accounts: they sign up at /signup, and you approve them on the people page (/admin/users).

6. Put it behind HTTPS

Port 8080 is plain HTTP. Before users outside a trusted network use it, put a reverse proxy in front with a real certificate. With Caddy, which gets the certificate by itself:

sudo apt-get install -y caddy          # Debian and Ubuntu
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
sudo systemctl reload caddy

Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go through the proxy. There is an nginx configuration too: deploy/proxy/.

When something does not work

What you see	What to do
`node add` stops at `join`	`sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443 on the control plane.
`node add` says the address is not in the certificate	Run `sudo llmgw server init --host <that address>`, then `node add` again.
A node says it cannot run vLLM	The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free.
A model shows `not placed`	`llmgw ctl explain-placement <model>` names each node and why it was not used.
A reply stops after 60 seconds behind nginx	Use the nginx configuration from the release. Its timeouts are set for slow models.
Anything else about a node	`sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you.

install.md: the same steps with every option explained.
fabrics.md: InfiniBand and NVLink.
onboarding.md: accounts, projects, budgets and keys.

7.2 KiB Raw Blame History