llmgw/docs/install.md

Installing llmgw from nothing

Written for whoever is standing up the cluster: you have servers, you have root on them, and nothing is installed yet. By the end there is a control plane, a node in the fleet, and a model answering requests.

It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond the package names. Everything here has been done on a real cluster except where this page says otherwise, and where it says otherwise it says so plainly.

What llmgw does not install

Read this first, because it is the difference between an afternoon and a week.

llmgw runs engines and routes requests. It does not install GPU drivers, CUDA, inference engines, or model weights. A node missing any of them still registers, still reports ready, and still fails every replica placed on it — so the useful order is to make a node able to serve before onboarding it, not after.

You provide	llmgw provides
NVIDIA driver and nvidia-smi	GPU discovery, classes, health
InfiniBand stack (rdma-core/OFED)	Fabric discovery and the invariant that a replica never spans one
vLLM, SGLang or llama.cpp	Launching them, the right flags, supervision
Model weights on disk	Placement, routing, accounting

The reason is boring and unavoidable: drivers and engines are platform decisions with kernel modules and version matrices attached, and distributing weights is its own problem — hundreds of gigabytes that want a shared filesystem, not a copy from somebody's laptop. It is a known gap, not an oversight.

0. Before you begin

Four machines is a reasonable first cluster: one control plane and three nodes. One of each also works.

The control plane needs no GPU. It is a single process holding a SQLite file; 2 GB of RAM and a few gigabytes of disk is generous. It must be reachable from every node on the agent port, and from your users on the client port.

Nodes need whatever the model needs, plus about 8 GB for the operating system and the agent. They dial the control plane; the control plane never dials them, so nodes need no inbound access except your own SSH.

Ports

Port	Who connects	Expose to
8080	clients, the web UI	your users, usually via a reverse proxy
7443	node agents	the node network only
9090	llmgw ctl, metrics, pprof	loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap

Decide now which address nodes will use to reach the control plane. It goes into a certificate and changing it later means reissuing one.

1. The binary

One static binary, no runtime dependencies, the same on every machine.

VERSION=v0.6.0    # check the releases page for the current tag
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64
curl -fsSL  https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version

arm64 and darwin builds are published alongside. They are built with CGO_ENABLED=0 deliberately, so a binary built today runs on a node whose libc is older than the builder's.

To build instead of download, you need Go 1.26 or later and nothing else:

git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw

2. The certificate authority

Agents authenticate to the control plane with mutual TLS, and a node's identity is taken from its certificate's common name. A public authority cannot supply these: Let's Encrypt and its peers stopped issuing certificates carrying the clientAuth usage, so nothing they hand out can authenticate an agent.

Run this wherever you intend to keep the authority. It talks to nothing and works on files, so it does not have to be a cluster machine — and there is an argument that it should not be.

sudo llmgw ctl cert init-ca --dir /etc/llmgw/pki

That writes ca.pem and ca-key.pem. Whoever holds ca-key.pem can mint any node's identity, so it stays here; ca.pem is copied to every machine and is not a secret.

Now the control plane's own certificate. Every address an agent might dial must appear in --host, bare IP addresses included — hostname verification reads subject alternative names and ignores the common name entirely, so a certificate naming the host only in its subject is rejected by every client:

sudo llmgw ctl cert issue-server llmgw-ctl \
  --host 10.0.0.140 --host llmgw-ctl.internal \
  --dir /etc/llmgw/pki --out /etc/llmgw

This covers the fleet-facing port only. The client-facing API is public and belongs behind an ordinary publicly trusted certificate, usually terminated by a reverse proxy in front of port 8080.

3. The control plane

The user and directories

sudo useradd --system --home /var/lib/llmgw --shell /usr/sbin/nologin llmgw
sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw
sudo install -d -m 0755 /etc/llmgw
sudo chown root:llmgw /etc/llmgw/ca.pem /etc/llmgw/server.pem /etc/llmgw/server-key.pem
sudo chmod 0640 /etc/llmgw/server-key.pem

The configuration

/etc/llmgw/server.yaml. This is the smallest file that is not a toy: mutual TLS on, authentication on, one model, and limits that apply to anyone nobody has given limits to.

listen: ":8080"
admin_listen: "127.0.0.1:9090"
agent_listen: ":7443"

agent_tls:
  ca: /etc/llmgw/ca.pem
  cert: /etc/llmgw/server.pem
  key: /etc/llmgw/server-key.pem

auth:
  enabled: true
  store: /var/lib/llmgw/llmgw.db
  signup_open: true
  # Applied to any key no scope gives a limit to. Without these, a project
  # created without limits is metered by nothing at all, and the first sign of
  # that is the bill or the outage.
  default_rpm: 60
  default_tpm: 200000

landing:
  contact: "platform@example.com"

# Start with none. Models are added while the cluster runs — see step 7 —
# and anything listed here is declarative: it is in the catalog on every start
# and cannot be removed without editing this file.
models: []

admin_listen is on loopback on purpose. It serves pprof, which can dump the process heap, and API keys sit in that heap in plaintext. To see the dashboard from elsewhere, promote an account and use /admin on the client port — do not move this listener.

The store file is created on first start; there is no separate migration step.

The service

sudo cp deploy/systemd/llmgw-server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llmgw-server
systemctl status llmgw-server

Your account

Accounts are created and approved from a shell on this machine, deliberately: having a shell here is what makes someone an administrator, and there are no roles to get subtly wrong.

sudo -u llmgw llmgw users create you@example.com --password 'change me now'
# prints a user id, usr-xxxxxxxx
sudo -u llmgw llmgw users promote usr-xxxxxxxx

promote is what grants /admin. Sign in at http://<control-plane>:8080/login, change the password, and issue yourself a key from the profile page.

Check it

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health   # 200, with an empty body
sudo -u llmgw llmgw ctl status

ctl status will report no nodes and no models, which is correct — nothing has been added yet. It is worth running now so its output is familiar before you need it.

4. Preparing a node

Do this before onboarding. A node missing its driver or engine registers healthy and fails on first placement, and the reason is three logs away.

The driver

sudo apt-get update
sudo apt-get install -y nvidia-driver-550   # or whatever your cards need
sudo reboot
nvidia-smi                                   # must print your GPUs

nvidia-smi must be on PATH for the user the agent runs as, which is root. Containerised driver installs often put it elsewhere; if so, note the path and set discovery.nvidia_smi in the agent configuration rather than moving it.

A node with no GPU at all is fine and is not an error. It reports zero cards and can host CPU profiles, which is a good way to try llmgw before committing hardware to it.

NVLink, if your GPUs have it

The published binary cannot see NVLink topology. NVML is the only source of it, its bindings need cgo, and the portable binary is built without cgo on purpose so it runs on a node whose libc is older than the builder's.

Without that topology, a tensor-parallel group takes the lowest free GPU indices. On an SXM board with NVSwitch that is correct, because every card is equidistant. On a PCIe box where NVLink bridges pair specific cards, a group straddling two bridges falls back to PCIe and runs several times slower — and nothing reports it, because from the outside it is simply a replica that is slow.

For those nodes, install the -nvml build published beside the portable one:

sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw

It is dynamically linked, so it needs a libc no older than the builder's and the NVIDIA driver present. Check it took effect with:

llmgw ctl nodes --json | grep -c nvlink_peers    # 0 means the topology is unknown

llmgw_gpu_nvlink_peers in the metrics says the same thing across a fleet: a flat zero everywhere means the topology is unknown rather than absent.

InfiniBand, if you have it

sudo apt-get install -y rdma-core ibverbs-utils infiniband-diags
ibstat            # must list your HCAs with State: Active
ls /sys/class/infiniband

The agent checks what you declare against what the node has. An adapter that is not there, or that has no ACTIVE port — physically linked but with no subnet manager, which survives a reboot and looks like a software problem — takes that fabric out of service on this node, loudly, in the log. Without that check the replica starts, passes its health checks, and hangs on its first all-reduce.

Fabric membership is configuration, not discovery, and there is no way around that: OpenSM's default subnet prefix is the same on every fabric, so a matching prefix proves nothing and a wrong guess produces a replica that starts, passes health checks, and then hangs on its first all-reduce. You name the fabric when you onboard the node, and two nodes are only in the same fabric if you said so.

An engine

Pick what you will actually serve. For a first cluster, llama.cpp is the least trouble and runs without a GPU.

# llama.cpp, from a release build
sudo apt-get install -y libgomp1          # it will not start without this
sudo mkdir -p /opt/llama.cpp
# unpack a llama.cpp release into /opt/llama.cpp
/opt/llama.cpp/llama-server --version

That libgomp1 line is not padding. A llama-server binary missing it exists, is executable, and cannot start — and the node reports ready throughout.

For vLLM, a virtualenv on the node:

sudo apt-get install -y python3-venv
sudo python3 -m venv /opt/venvs/vllm
sudo /opt/venvs/vllm/bin/pip install vllm
/opt/venvs/vllm/bin/vllm --version

Check the engine still takes the flags llmgw builds for it, which drift between releases:

llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm

Weights

Put them where the node can read them. There is no distribution mechanism yet; scp, a shared filesystem, or the engine's own downloader are all valid, and which you want depends on how many nodes you have.

sudo install -d -o root -g root /var/lib/llmgw-models
# then place your .gguf or model directory there

For a first model, something small: a 3B at q4_k_m is about 2 GB, runs on CPU, and answers fast enough to prove the whole path works.

Letting llmgw fetch them

The simple arrangement, and the default: every node keeps its own copy. Point a model's source at a file to download and the agent fetches it once, before the engine is launched:

source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
# or an ordinary URL
source: https://models.internal/qwen2.5-coder-3b-q4_k_m.gguf

Eight shards of one replica starting together produce one download, not eight. An interrupted fetch resumes rather than starting over, and a partial file is never visible under the model's name — so a half-downloaded model cannot be picked up by the next scan and handed to an engine.

Three kinds of source, and only one of them is fetched:

source	What happens
/var/lib/llmgw-models/x.gguf	Checked, never copied. However it got there.
hf:org/repo/file.gguf, https://…	Fetched once into the cache, then reused
Qwen/Qwen2.5-7B-Instruct	Left alone — vLLM and SGLang download repositories themselves

Which means the exotic arrangements need no configuration at all. Weights on a Lustre mount, or an image exported over the fabric with SRP or NVMe-oF, are simply present, and present means nothing is fetched.

Telling llmgw where they are

Optional, and unset by default. Naming the directories in the agent's model_dirs lets a node report which models it can see, and placement then prefers a node that already holds one over a node that must first move tens of gigabytes:

model_dirs:
  - /var/lib/llmgw-models
  - /mnt/lustre/models

llmgw does not manage that storage and does not care how the files arrived. A Lustre or NFS mount, an image exported over the fabric with SRP or NVMe-oF, local NVMe, or an scp all look identical to it. A node with nothing configured reports nothing and is treated exactly as every node was before this existed.

It matters most when your fabrics are not connected to each other. Weights in ib-a are not weights in ib-b: the only path between them is the management network, which is the slow one. llmgw ctl status shows what each fabric holds, so a model seeded into one fabric does not quietly leave the rest of the cluster unable to serve it.

How much disk a node needs follows from the models rather than a rule of thumb. llmgw ctl model plan <repo> prints the sizes; as a starting point, a 70B at bf16 is 141 GB and the same model at q4 is 43 GB.

5. Adding the node

Two halves, because they need different access: building the machine needs the hypervisor, and issuing its identity needs the certificate authority. Where one person holds both — the common case — one command does it:

llmgw-add-node \
  --name llmgw-node01 --ip 10.0.0.141/24 \
  --server https://10.0.0.140:7443 \
  --cores 16 --memory 131072 --disk 200 \
  --fabric ib-a:mlx5_0,mlx5_1 \
  --llamacpp-bin /opt/llama.cpp/llama-server

The VM name, the hostname, the node id and the certificate's common name are all the same string. They do not have to be, and every time they are not, somebody spends an afternoon working out which of the four a message refers to.

For a machine that already exists — bare metal, or a VM you built yourself — skip provisioning and issue the identity only:

llmgw-onboard-node \
  --node llmgw-node01 --addr 10.0.0.141 \
  --server https://10.0.0.140:7443 \
  --fabric ib-a:mlx5_0,mlx5_1 \
  --llamacpp-bin /opt/llama.cpp/llama-server

Onboarding needs SSH from the CA host to the node. It waits for SSH rather than failing on it, and learns the host key of a machine it has not seen — a new machine spends its first minute regenerating host keys and restarting sshd.

It also checks that a configured engine actually runs, and says so loudly when it does not. That check exists because the failure is otherwise invisible.

Within a few seconds:

llmgw ctl nodes

NODE          STATE  GPUS      MEM   CLASSES   FABRICS  REPLICAS  LAST SEEN
llmgw-node01  ready  8/8 free  503G  h100-80g  ib-a     0         2s

If it does not appear, the control plane's log says why. A certificate whose common name does not match the claimed node id is refused by design — that check is what makes a node's identity something it proves rather than something it picks.

Repeat for each node. Nodes in the same fabric must be given the same --fabric name; nodes with no InfiniBand take no --fabric at all and are each their own island, which is fine for single-node models.

6. Setting the node's CPU model, if it is a VM

Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2 from the guest. Inference then falls back to scalar code and is not slightly slower but unusable. Measured on one 8-core VM, same weights, same threads, the only change being --cpu host:

Qwen2.5-Coder	kvm64	host
1.5B q4_K_M	5.80	39.09 tok/s
3B q4_K_M	2.90	20.51
7B q4_K_M	1.33	9.88

llmgw-provision-node sets --cpu host for you. A VM you built yourself probably did not, and nothing about it looks broken — every number is simply small. Check with grep -qw avx2 /proc/cpuinfo on the node.

7. Adding a model

Write the definition. This is the same shape a model takes in server.yaml, so an entry moves between the two by being copied:

# qwen-coder-3b.yaml
name: qwen2.5-coder-3b-instruct
aliases: [coder]
source: /var/lib/llmgw-models/qwen2.5-coder-3b-instruct-q4_k_m.gguf
routing_profile: chat
# What a client may use in one request. Worth stating: llama.cpp divides
# --ctx-size across its parallel slots, so a pool of 65536 over two slots is
# 32768 each, and nothing else will tell you that.
context_length: 32768
max_output: 4096
scaling: { min: 1, max: 1 }
profiles:
  - name: cpu
    engine: llamacpp
    weight: 1
    priority: 100
    startup_timeout: 5m
    min_memory_gb: 8
    args: ["--ctx-size", "65536"]

llmgw ctl model add -f qwen-coder-3b.yaml
llmgw ctl models

MODEL                      SERVING  REPLICAS  STATUS   ALIASES
qwen2.5-coder-3b-instruct  yes      1/1       ready    coder

No restart, and it survives one: models added this way are stored, while anything in server.yaml is declarative and returns on every start.

For a GPU model, name the class and the width instead:

profiles:
  - name: tp2
    engine: vllm
    gpu_class: h100-80g    # as `llmgw ctl nodes` reports it
    tp: 2
    priority: 100

If it does not place, ask:

llmgw ctl explain-placement qwen2.5-coder-3b-instruct

which names every candidate and why each was rejected, rather than leaving you to infer it.

8. The first request

curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "coder", "messages": [{"role":"user","content":"hello"}]}'

Or open /chat in a browser and use it without a key at all — the page carries your session, and the request still goes through admission, rate limits, budget and accounting exactly as an API call does.

When something does not work

The node is ready but every replica fails. Almost always a missing engine or weights. llmgw ctl replicas shows the shard's error and the engine's last output, which is the actual diagnosis.

A model will not place. llmgw ctl explain-placement <model> names each node and the reason: wrong GPU class, not enough free cards, too little memory, no fabric, a label it does not carry.

Everything is slow. On a VM, check AVX2 (step 6). On CPU, check the quantisation — the difference between q8_0 and q4_k_m is roughly a factor of two. llmgw ctl status shows how full the cluster is.

Tool calls come back as text. The model made the call and the engine did not parse it. llmgw ctl tool-check --endpoint http://<node>:<port> distinguishes that from a model ignoring its tools, which has a different fix.

Requests fail at exactly sixty seconds. timeouts.response_header is a liveness check for streaming and a total generation budget for non-streaming requests. On a slow model, set timeouts.generation higher.

What to read next

• docs/node-onboarding.md — adding nodes in more detail, and what onboarding deliberately does not do
• docs/onboarding.md — giving people accounts and keys
• llmgw-plan.md — why the system is shaped the way it is

Known gaps, so you find them here rather than at 2am

• Weights are yours to distribute. No cache, no fetch, no progress.
• Engines are yours to install. Onboarding verifies one runs; it installs nothing.
• One control plane. No HA yet; it is a single process with a SQLite file. Back up /var/lib/llmgw/llmgw.db.
• Multi-node replicas have not been run on real InfiniBand. The scheduling, the fabric invariant and the rendezvous are exercised against simulated hardware only. Single-node placement on real GPUs is the well-trodden path; if you are the first to run a TP group across real cards, expect to find something.