ilker/llmgw
2
0 Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 95 Wiki Activity

Docs: one way in — quick-install starts at the setup screens, quickstart becomes development, install.md no longer says engines are yours to install

Browse Source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 17:32:54 +02:00
1 parent 4979e08d64
commit 6abd510628
5 files changed
+103 -72
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+44 -29
View File
@@ -8,15 +8,17 @@ OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
## Getting started
Starting from nothing — no packages installed, no drivers, no nodes — follow
[docs/install.md](docs/install.md): it builds a control plane, adds a node and
serves a model, and says plainly which parts llmgw does not install for you.
On the machine that will be the control plane, with the release binary
installed:
| Guide | For |
|---|---|
| [docs/install.md](docs/install.md) | Standing up a cluster from scratch |
| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists |
| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys |
```sh
sudo llmgw
```
The setup screens install one machine or three (high availability), check
them first, and leave a model answering. After that the same command opens
the console. [docs/quick-install.md](docs/quick-install.md) has the six steps
from empty servers to a first request; everything else is below.
## Status
@@ -82,37 +84,50 @@ them. On such a machine, run `llmgw doctor --json` first and keep the report;
## Documentation
- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in
six steps: start here
- [docs/install.md](docs/install.md) — the same from nothing, with every option
explained: control plane, nodes, engines, weights, first request
- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source
Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
**Installing**
- [docs/quick-install.md](docs/quick-install.md) — from empty servers to a
first request, in six steps: start here
- [docs/install.md](docs/install.md) — the same with every option explained,
and running the cluster after: monitoring, notifications, single sign-on,
the firewall, upgrades, backups
- [docs/node-onboarding.md](docs/node-onboarding.md) — adding nodes to a
cluster that exists
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
when a replica spanning nodes will not run
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
how many of each, and where the weights should live
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
what leaves the machine; written for a security review
**Running it**
- [docs/console.md](docs/console.md) — the console that bare `llmgw` opens:
every view, and the command each of its actions runs
- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
policies, detectors, a judge model, and where each applies
- [docs/onboarding.md](docs/onboarding.md) — accounts, keys, limits, staff
roles and single sign-on
- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
policies, detectors, a judge model, and where each applies
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
how many of each, and where the weights should live
**Reference**
- [docs/reference.md](docs/reference.md) — every configuration key, generated
from the structs and checked by a test, so it cannot describe a setting that
does not exist or miss one that does
- [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest
and most useful thing to build around llmgw
- [deploy/proxy/](deploy/proxy/) — Caddy and nginx in front of the client
port, with streaming and the forwarded headers right
- [deploy/ha/](deploy/ha/) — running two control planes, and the one health
check that has to be right
- [deploy/monitoring/](deploy/monitoring/) — Prometheus rules, Alertmanager
with receiver templates, and Grafana dashboards
- [docs/node-onboarding.md](docs/node-onboarding.md),
[docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
what leaves the machine; written for a security review
- [deploy/](deploy/) — [a mirror](deploy/mirror/) on its own machine,
[Caddy and nginx](deploy/proxy/) in front of the client port, [two control
planes](deploy/ha/) behind a balancer, and [Prometheus, Alertmanager and
Grafana](deploy/monitoring/)
**Working on llmgw**
- [docs/development.md](docs/development.md) — built from source on one
workstation, no systemd and no `sudo`
- [llmgw-plan.md](llmgw-plan.md) — the design; [BACKLOG.md](BACKLOG.md) —
what is next
## Building
docs/quickstart.md → docs/development.md
+4 -3
View File
@@ -1,8 +1,9 @@
# Quickstart: one machine, one model, ten minutes
# Development: llmgw from source on one machine
Written for a developer who wants to see llmgw answer a request on a single
workstation, built from source, with no systemd and no `sudo`. **To install
llmgw on servers, read [quick-install.md](quick-install.md) instead.**
workstation, built from source, with no systemd and no `sudo` — for working on
llmgw itself. **To install llmgw on servers, read
[quick-install.md](quick-install.md) instead.**
What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
of a real model served by llama.cpp.
docs/install.md
+20 -20
View File
@@ -13,34 +13,32 @@ this page says otherwise, and where it says otherwise it says so plainly.
Read this first, because it is the difference between an afternoon and a week.
llmgw runs engines and routes requests. It does **not** install GPU drivers,
CUDA, or the inference engines themselves. A node missing any of them still
registers, still reports `ready`, and still fails every replica placed on it —
so the useful order is to make a node able to serve *before* onboarding it, not
after.
llmgw installs itself, its engines and its store. It does **not** install the
GPU driver or the InfiniBand stack: those are kernel modules with version
matrices attached, and a platform decision. A node missing them still
registers and reports `ready`, and fails every replica that needs the GPU or
the fabric — so make a node able to serve *before* onboarding it.
`llmgw doctor` on the node says what is missing and the command that fixes it.
| You provide | llmgw provides |
| --- | --- |
| NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
| InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision |
| Model weights, from somewhere | Fetching them, placement, routing, accounting |
| Nothing — the nodes install them | llama.cpp, vLLM (and SGLang when asked), with the right build for each node's driver and GPU: [Engines](#engines) |
| Which models to run | The mirror that fetches them once, placement, routing, accounting |
The reason is boring and unavoidable: drivers and engines are platform decisions
with kernel modules and version matrices attached.
Weights are the one item on that list llmgw does handle. It includes a mirror —
step 3 — which holds the files on your own network and hands them to the nodes,
so getting weights onto every machine is not a problem you solve before
starting. What llmgw will not do is choose the models for you.
Weights are handled too. llmgw includes a mirror — step 3 — which holds the
files on your own network and hands them to the nodes, so getting weights onto
every machine is not a problem you solve before starting. What llmgw will not do is choose the models for you.
## 0. Before you begin
Four machines is a reasonable first cluster: one control plane and three nodes.
One of each also works.
**The control plane** needs no GPU. It is a single process holding a SQLite
file; 2 GB of RAM and a few gigabytes of disk is generous. It must be reachable
**The control plane** needs no GPU: one machine, holding its store in SQLite
or PostgreSQL, or three for high availability, with the store in PostgreSQL replicated under
Patroni. 2 GB of RAM and a few gigabytes of disk each is generous. It must be reachable
from every node on the agent port, and from your users on the client port.
**Nodes** need whatever the model needs, plus about 8 GB for the operating
@@ -67,9 +65,11 @@ a certificate: the machine's own addresses are found for you, and any other name
One static binary, no runtime dependencies, the same on every machine.
```
VERSION=v0.8.0 # check the releases page for the current tag
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64
curl -fsSL https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
| grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version
```
@@ -1101,7 +1101,7 @@ llmgw keys memory --project default --off
Without authentication, `threads.store` and `threads.retain_hours` in
`server.yaml` keep both conversations and notes in one shared pool. That is for
one trusted machine only; see [quickstart.md](quickstart.md).
one trusted machine only; see [development.md](development.md).
## Two control planes
docs/quick-install.md
+33 -18
View File
@@ -8,10 +8,10 @@ explains every choice.
| | |
|---|---|
| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
| **root** | on the control plane, and `sudo` without a password on every node. |
| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. |
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
@@ -35,26 +35,38 @@ install it.
## 2. Set up the control plane
```
sudo llmgw server init
sudo llmgw
```
It asks for your email and a few choices, then sets up everything on this
machine, in a few minutes. In a script, give them as flags instead:
`sudo llmgw server init --admin you@example.com`.
With nothing installed, `llmgw` opens the setup screens. They ask whether
this is **one machine or three** (high availability), and for three, the
other two: address, user, and an ssh key, the ssh agent or a password. They
check every machine before installing anything — system, disk, clock, ports,
internet, whether the machines reach one another — show the plan, and
install:
- the service, its certificates and its configuration;
- your administrator account. **The password is printed once. Copy it now.**
- the service, its certificates and its configuration, and a firewall for
its internal ports;
- with three machines, the store replicated across them (PostgreSQL under
Patroni);
- your administrator account. **The password is shown once. Copy it now.**
- a model mirror, which downloads model files once for the whole cluster;
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
Run again later, `sudo llmgw` is the console: the cluster's state and
everything you can do to it ([console.md](console.md)). Every step below is
also there, under **Do**.
**In a script**, the same for one machine:
```
sudo llmgw server init --admin you@example.com
```
If nodes will reach this machine by a name or address it does not know itself
(a DNS name, a NAT address), add `--host` for each:
```
sudo llmgw server init --admin you@example.com --host llm.example.com
```
It is safe to run again. It keeps what exists and finishes what is missing.
`--host llm.example.com`. It is safe to run again: it keeps what exists and
finishes what is missing.
**The mirror on its own machine**, when it wants a larger disk than the
control plane has: first on that machine, with llmgw installed as in step 1,
@@ -75,7 +87,7 @@ the account page.
## 3. Add the GPU nodes
On the control plane:
In the console, **Do → Add a node**. Or on the control plane:
```
sudo llmgw node add ubuntu@10.0.0.21
@@ -197,6 +209,9 @@ through the proxy. There is an nginx configuration too:
## Next
- [install.md](install.md): the same steps with every option explained.
- [install.md](install.md): the same steps with every option explained, and
running the cluster after: monitoring, notifications, single sign-on,
backups, upgrades.
- [console.md](console.md): everything the console does.
- [fabrics.md](fabrics.md): InfiniBand and NVLink.
- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.
internal/cli/tui_docs.go
+2 -2
View File
@@ -25,8 +25,8 @@ type docTopic struct {
}
// docOrder is the documents in the order someone new reads them.
var docOrder = []string{"console", "quickstart", "quick-install", "install", "node-onboarding", "onboarding",
"choosing-models", "fabrics", "facility-aware-placement", "data", "reference"}
var docOrder = []string{"console", "quick-install", "install", "node-onboarding", "onboarding",
"guardrails", "choosing-models", "fabrics", "facility-aware-placement", "data", "reference", "development"}
func docTopics() []docTopic {
read := func(path string) func() string {