Self-hosted LLM gateway · {{.Version}}

One endpoint in front of your own GPUs.

llmgw runs vLLM, SGLang and llama.cpp across your servers and serves them behind one OpenAI-compatible API. It knows which GPUs share an NVLink domain or an InfiniBand fabric, and a model split across machines never straddles two.

Download{{with .Release}} {{.}}{{end}} Install guide

$ sudo llmgw server init --admin you@example.com

A replica spanning nodes goes into one NVLink domain if one has room, otherwise onto one InfiniBand fabric. Two fabrics that cannot see each other are never bridged: the all-reduce would hang.

What it does

One static binary. It runs as the control plane, as the agent on every GPU node, and as the weights mirror.

OpenAI API, your hardware

Point any OpenAI client at it. Only the base URL changes. Streaming, tool calls, /v1/models and /v1/responses work as your client expects.

Placement that knows the fabric

Agents report GPU class, memory, NVLink peers and InfiniBand ports. Replicas go where they fit, including ones larger than one machine, on the GPUs with the most NVLink pairs.

Models load when asked for

Keep more models in the catalog than your GPUs hold. The first request starts one, the chat page shows it loading, and it stops again after the idle timeout.

Keys, budgets, rate limits

Organisations, teams and projects, each with limits. Priority bands decide who waits when the cluster is full. Managers see their organisation's spend; people see their own.

Editors connect from the account page

Pick Continue, Codex or Cline and a model. The page makes a key and hands over the config, filled in, or a one-line command that writes it.

A chat page for quick questions

Markdown and maths rendered, stop mid-reply, tokens per second and time to first token, a system prompt and a temperature. The gateway keeps no conversations.

A mirror you browse

Explore what is trending on Hugging Face, read a model card under its row, and download once for the whole cluster. Only what your nodes can run is shown by default.

Nodes install their own engines

llama.cpp on every node; vLLM on every GPU node that can run it, from PyPI with every file hash-checked. A node that cannot run it says why.

Upgrades that keep models running

Upgrade the control plane, then the agents. A restarted agent adopts the engines it was running instead of reloading them, so replicas stay up throughout.

A dashboard with nothing to set up

/admin shows capacity per GPU class, CPU and memory per node, who is using what, each node's page and logs. It fits a phone. Prometheus is optional.

InfiniBand and NVLink, several of each

A site may have independent InfiniBand fabrics, eight-GPU HGX boards and NVL72 racks side by side. llmgw tells them apart by what the adapters report, not by what a config file calls them.

Fabrics and NVLink guide

One NVLink domain

NVL72 or NVLink Switch. Every card joined, IMEX up. Shards get NCCL_MNNVL_ENABLE=1.

One InfiniBand or RoCE fabric

Nodes under one leaf first, nodes with a degraded cable last.

Never across two

Two fabrics given one name are placed as two, and the cluster page says so.

Check a node in one command

$ sudo llmgw doctor
FAIL  nvlink  this machine has NVSwitch but nvidia-fabricmanager is inactive
              → systemctl enable --now nvidia-fabricmanager
WARN  fabric  nvidia_peermem is not loaded: GPUDirect RDMA is off
              → modprobe nvidia_peermem
OK    gpu     8 GPUs: 8× NVIDIA H100 80GB HBM3
OK    nvlink  NVLink groups: [0 1 2 3 4 5 6 7]
OK    fabric  mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0
OK    fabric  subnet managers: 0x…f0c0 (master, priority 15), 0x…f1d4 (standby, priority 14)
OK    agent   NCCL bootstraps over ens3 (10.1.0.21)

Subnet managers, watchedMaster, priority, standbys and failovers per fabric. A fabric with no standby is flagged. llmgw does not run one.
A map of each fabric/admin/fabrics draws switches in tiers, cables coloured by rate, degraded ones in red, and live traffic on your adapters.
NVLink domains as racks/admin/nvlink shows which trays are joined, whose IMEX is down, which link runs degraded.
Ports re-read every heartbeatA port that goes down takes its fabric off the node within seconds. Counters and rates are exported to Prometheus.

From nothing to a first request

Debian, Ubuntu or RHEL 10. NVIDIA drivers and the InfiniBand stack are yours to install; llmgw does the rest.

The full guide covers the mirror, GPUs and what to do when something does not work.

Put the binary on every machine

Static, no dependencies, the same file everywhere.

VERSION={{or .Release "vX.Y.Z"}}{{if not .Release}}   # from the releases page{{end}}
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
curl -fsSL  $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw

Set up the control plane

Creates the user, the certificate authority, the certificate, server.yaml and your administrator account, then starts the service. Safe to run again. Add --host for any name nodes use that the machine cannot see itself.
```
sudo llmgw server init --admin you@example.com
```

Add a node

From the control plane, which holds the authority. The node gets its certificate, the binary and the agent, and installs its engine by itself.

deploy/bootstrap/onboard-node.sh --node gpu01 --addr 10.0.0.141 \
  --server https://10.0.0.140:7443 --fabric ib-a:mlx5_0,mlx5_1

$ llmgw ctl nodes
NODE   STATE  GPUS      MEM   CLASSES   FABRICS  REPLICAS  LAST SEEN
gpu01  ready  8/8 free  503G  h100-80g  ib-a     0         2s

Add a model and send a request

No restart. If it does not place, llmgw ctl explain-placement names every node and why it was turned down.

llmgw ctl model add -f qwen-coder-3b.yaml

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "coder", "messages": [{"role": "user", "content": "hello"}]}'

Behind a reverse proxy

Port 8080 is plain HTTP. Put it behind Caddy or nginx with a public certificate. Only 8080 goes through the proxy: agents reach 7443 directly, because it is mutual TLS with the cluster's own authority.

Bind llmgw to loopback when the proxy is on the same machine: listen: "127.0.0.1:8080".

Caddy nginx

Obtains and renews its own certificate. Replace llm.example.com. Caddyfile

{{.Caddy}}

Certificate from certbot --nginx. Works on nginx 1.18 and later. nginx.conf

{{.Nginx}}

Streams are not bufferedTokens reach the client as they are generated, not in one burst at the end.
The read timeout outlasts the modelnginx's 60-second default turns a slow first token into a 504.
Forwarded headers are set, not passed onllmgw marks session cookies Secure from X-Forwarded-Proto and shows X-Forwarded-Host as the base URL.
Body limit matches the router32 MiB, so an oversized request is refused before it is uploaded.

Download

Built by CI from the tag and checked against checksums.txt. Every release is on the releases page with its changelog.

{{.Release}}{{with .Date}}Released {{.}}{{end}}

For	File	Notes
Linux x86-64	`llmgw-{{$.Release}}-linux-amd64`	Static. Runs on any libc.
Linux ARM64	`llmgw-{{$.Release}}-linux-arm64`	Static.
Linux x86-64, NVLink	`llmgw-{{$.Release}}-linux-amd64-nvml`	Sees NVLink topology through NVML. For PCIe boxes with NVLink bridges. Needs the driver.
macOS Apple silicon	`llmgw-{{$.Release}}-darwin-arm64`	For `llmgw ctl` and trying it on a laptop.
macOS Intel	`llmgw-{{$.Release}}-darwin-amd64`
Caddy	`Caddyfile`	Reverse proxy for port 8080.
nginx	`nginx.conf`	Reverse proxy for port 8080.
Checksums	`checksums.txt`	SHA-256 of every file above.

This is a development build ({{.Version}}). Take a release from the releases page, where every file is listed with its checksum.

Where it stands

Pre-1.0: configuration formats may change between minor versions. This is what has run on real hardware, and what has not yet.

Run on real clusters

Single-node replicas on real GPUs and on CPU
On-demand loading and idle shutdown
Keys, budgets, rate limits and priority bands
Rolling upgrades with no dropped replicas
SQLite and PostgreSQL stores
Chat, editor setup and the Responses API against a six-node test cluster

Not yet proven

InfiniBand and NVLink on a real site: written and tested against the real output of sminfo, ibnetdiscover and nvidia-smi, but not yet run on the hardware. Run llmgw doctor --json on a few nodes first.
Two control planes over long periods
Nodes do not share downloaded weights; the mirror keeps each download local