From 6abd51062843dd8af5c59451df7f3f5bd62f6bb9 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 4 Oct 2026 17:32:54 +0200 Subject: [PATCH] =?UTF-8?q?Docs:=20one=20way=20in=20=E2=80=94=20quick-inst?= =?UTF-8?q?all=20starts=20at=20the=20setup=20screens,=20quickstart=20becom?= =?UTF-8?q?es=20development,=20install.md=20no=20longer=20says=20engines?= =?UTF-8?q?=20are=20yours=20to=20install?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 73 ++++++++++++++++---------- docs/{quickstart.md => development.md} | 7 +-- docs/install.md | 40 +++++++------- docs/quick-install.md | 51 +++++++++++------- internal/cli/tui_docs.go | 4 +- 5 files changed, 103 insertions(+), 72 deletions(-) rename docs/{quickstart.md => development.md} (98%) diff --git a/README.md b/README.md index e708c47..248b969 100644 --- a/README.md +++ b/README.md @@ -8,15 +8,17 @@ OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md). ## Getting started -Starting from nothing — no packages installed, no drivers, no nodes — follow -[docs/install.md](docs/install.md): it builds a control plane, adds a node and -serves a model, and says plainly which parts llmgw does not install for you. +On the machine that will be the control plane, with the release binary +installed: -| Guide | For | -|---|---| -| [docs/install.md](docs/install.md) | Standing up a cluster from scratch | -| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists | -| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys | +```sh +sudo llmgw +``` + +The setup screens install one machine or three (high availability), check +them first, and leave a model answering. After that the same command opens +the console. [docs/quick-install.md](docs/quick-install.md) has the six steps +from empty servers to a first request; everything else is below. ## Status @@ -82,37 +84,50 @@ them. On such a machine, run `llmgw doctor --json` first and keep the report; ## Documentation -- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in - six steps: start here -- [docs/install.md](docs/install.md) — the same from nothing, with every option - explained: control plane, nodes, engines, weights, first request -- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source +Every page is also in the binary: `llmgw docs`, or **Manual** in the console. + +**Installing** + +- [docs/quick-install.md](docs/quick-install.md) — from empty servers to a + first request, in six steps: start here +- [docs/install.md](docs/install.md) — the same with every option explained, + and running the cluster after: monitoring, notifications, single sign-on, + the firewall, upgrades, backups +- [docs/node-onboarding.md](docs/node-onboarding.md) — adding nodes to a + cluster that exists - [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what llmgw needs of them, checking a node with `llmgw doctor`, and what to look at when a replica spanning nodes will not run -- [docs/choosing-models.md](docs/choosing-models.md) — which models to run, - how many of each, and where the weights should live -- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and - what leaves the machine; written for a security review + +**Running it** + - [docs/console.md](docs/console.md) — the console that bare `llmgw` opens: every view, and the command each of its actions runs -- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers: - policies, detectors, a judge model, and where each applies - [docs/onboarding.md](docs/onboarding.md) — accounts, keys, limits, staff roles and single sign-on +- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers: + policies, detectors, a judge model, and where each applies +- [docs/choosing-models.md](docs/choosing-models.md) — which models to run, + how many of each, and where the weights should live + +**Reference** + - [docs/reference.md](docs/reference.md) — every configuration key, generated from the structs and checked by a test, so it cannot describe a setting that does not exist or miss one that does -- [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest - and most useful thing to build around llmgw -- [deploy/proxy/](deploy/proxy/) — Caddy and nginx in front of the client - port, with streaming and the forwarded headers right -- [deploy/ha/](deploy/ha/) — running two control planes, and the one health - check that has to be right -- [deploy/monitoring/](deploy/monitoring/) — Prometheus rules, Alertmanager - with receiver templates, and Grafana dashboards -- [docs/node-onboarding.md](docs/node-onboarding.md), - [docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people +- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and + what leaves the machine; written for a security review +- [deploy/](deploy/) — [a mirror](deploy/mirror/) on its own machine, + [Caddy and nginx](deploy/proxy/) in front of the client port, [two control + planes](deploy/ha/) behind a balancer, and [Prometheus, Alertmanager and + Grafana](deploy/monitoring/) + +**Working on llmgw** + +- [docs/development.md](docs/development.md) — built from source on one + workstation, no systemd and no `sudo` +- [llmgw-plan.md](llmgw-plan.md) — the design; [BACKLOG.md](BACKLOG.md) — + what is next ## Building diff --git a/docs/quickstart.md b/docs/development.md similarity index 98% rename from docs/quickstart.md rename to docs/development.md index 1e8754b..8a8a07c 100644 --- a/docs/quickstart.md +++ b/docs/development.md @@ -1,8 +1,9 @@ -# Quickstart: one machine, one model, ten minutes +# Development: llmgw from source on one machine Written for a developer who wants to see llmgw answer a request on a single -workstation, built from source, with no systemd and no `sudo`. **To install -llmgw on servers, read [quick-install.md](quick-install.md) instead.** +workstation, built from source, with no systemd and no `sudo` — for working on +llmgw itself. **To install llmgw on servers, read +[quick-install.md](quick-install.md) instead.** What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front of a real model served by llama.cpp. diff --git a/docs/install.md b/docs/install.md index c91daa0..2c41d1f 100644 --- a/docs/install.md +++ b/docs/install.md @@ -13,34 +13,32 @@ this page says otherwise, and where it says otherwise it says so plainly. Read this first, because it is the difference between an afternoon and a week. -llmgw runs engines and routes requests. It does **not** install GPU drivers, -CUDA, or the inference engines themselves. A node missing any of them still -registers, still reports `ready`, and still fails every replica placed on it — -so the useful order is to make a node able to serve *before* onboarding it, not -after. +llmgw installs itself, its engines and its store. It does **not** install the +GPU driver or the InfiniBand stack: those are kernel modules with version +matrices attached, and a platform decision. A node missing them still +registers and reports `ready`, and fails every replica that needs the GPU or +the fabric — so make a node able to serve *before* onboarding it. +`llmgw doctor` on the node says what is missing and the command that fixes it. | You provide | llmgw provides | | --- | --- | | NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health | | InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one | -| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision | -| Model weights, from somewhere | Fetching them, placement, routing, accounting | +| Nothing — the nodes install them | llama.cpp, vLLM (and SGLang when asked), with the right build for each node's driver and GPU: [Engines](#engines) | +| Which models to run | The mirror that fetches them once, placement, routing, accounting | -The reason is boring and unavoidable: drivers and engines are platform decisions -with kernel modules and version matrices attached. - -Weights are the one item on that list llmgw does handle. It includes a mirror — -step 3 — which holds the files on your own network and hands them to the nodes, -so getting weights onto every machine is not a problem you solve before -starting. What llmgw will not do is choose the models for you. +Weights are handled too. llmgw includes a mirror — step 3 — which holds the +files on your own network and hands them to the nodes, so getting weights onto +every machine is not a problem you solve before starting. What llmgw will not do is choose the models for you. ## 0. Before you begin Four machines is a reasonable first cluster: one control plane and three nodes. One of each also works. -**The control plane** needs no GPU. It is a single process holding a SQLite -file; 2 GB of RAM and a few gigabytes of disk is generous. It must be reachable +**The control plane** needs no GPU: one machine, holding its store in SQLite +or PostgreSQL, or three for high availability, with the store in PostgreSQL replicated under +Patroni. 2 GB of RAM and a few gigabytes of disk each is generous. It must be reachable from every node on the agent port, and from your users on the client port. **Nodes** need whatever the model needs, plus about 8 GB for the operating @@ -67,9 +65,11 @@ a certificate: the machine's own addresses are found for you, and any other name One static binary, no runtime dependencies, the same on every machine. ``` -VERSION=v0.8.0 # check the releases page for the current tag -curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64 -curl -fsSL https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing +VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \ + | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4) +U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION +curl -fsSLO $U/llmgw-$VERSION-linux-amd64 +curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw llmgw --version ``` @@ -1101,7 +1101,7 @@ llmgw keys memory --project default --off Without authentication, `threads.store` and `threads.retain_hours` in `server.yaml` keep both conversations and notes in one shared pool. That is for -one trusted machine only; see [quickstart.md](quickstart.md). +one trusted machine only; see [development.md](development.md). ## Two control planes diff --git a/docs/quick-install.md b/docs/quick-install.md index cc77db5..66a131c 100644 --- a/docs/quick-install.md +++ b/docs/quick-install.md @@ -8,10 +8,10 @@ explains every choice. | | | |---|---| -| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. | -| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). | -| **root** | on the control plane, and `sudo` without a password on every node. | -| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. | +| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each. | +| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. | +| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. | +| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). | | **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. | You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that @@ -35,26 +35,38 @@ install it. ## 2. Set up the control plane ``` -sudo llmgw server init +sudo llmgw ``` -It asks for your email and a few choices, then sets up everything on this -machine, in a few minutes. In a script, give them as flags instead: -`sudo llmgw server init --admin you@example.com`. +With nothing installed, `llmgw` opens the setup screens. They ask whether +this is **one machine or three** (high availability), and for three, the +other two: address, user, and an ssh key, the ssh agent or a password. They +check every machine before installing anything — system, disk, clock, ports, +internet, whether the machines reach one another — show the plan, and +install: -- the service, its certificates and its configuration; -- your administrator account. **The password is printed once. Copy it now.** +- the service, its certificates and its configuration, and a firewall for + its internal ports; +- with three machines, the store replicated across them (PostgreSQL under + Patroni); +- your administrator account. **The password is shown once. Copy it now.** - a model mirror, which downloads model files once for the whole cluster; - this machine as a first node, and a small model (Qwen2.5 0.5B) on it. +Run again later, `sudo llmgw` is the console: the cluster's state and +everything you can do to it ([console.md](console.md)). Every step below is +also there, under **Do**. + +**In a script**, the same for one machine: + +``` +sudo llmgw server init --admin you@example.com +``` + If nodes will reach this machine by a name or address it does not know itself (a DNS name, a NAT address), add `--host` for each: - -``` -sudo llmgw server init --admin you@example.com --host llm.example.com -``` - -It is safe to run again. It keeps what exists and finishes what is missing. +`--host llm.example.com`. It is safe to run again: it keeps what exists and +finishes what is missing. **The mirror on its own machine**, when it wants a larger disk than the control plane has: first on that machine, with llmgw installed as in step 1, @@ -75,7 +87,7 @@ the account page. ## 3. Add the GPU nodes -On the control plane: +In the console, **Do → Add a node**. Or on the control plane: ``` sudo llmgw node add ubuntu@10.0.0.21 @@ -197,6 +209,9 @@ through the proxy. There is an nginx configuration too: ## Next -- [install.md](install.md): the same steps with every option explained. +- [install.md](install.md): the same steps with every option explained, and + running the cluster after: monitoring, notifications, single sign-on, + backups, upgrades. +- [console.md](console.md): everything the console does. - [fabrics.md](fabrics.md): InfiniBand and NVLink. - [onboarding.md](onboarding.md): accounts, projects, budgets and keys. diff --git a/internal/cli/tui_docs.go b/internal/cli/tui_docs.go index ee689bf..4f76b98 100644 --- a/internal/cli/tui_docs.go +++ b/internal/cli/tui_docs.go @@ -25,8 +25,8 @@ type docTopic struct { } // docOrder is the documents in the order someone new reads them. -var docOrder = []string{"console", "quickstart", "quick-install", "install", "node-onboarding", "onboarding", - "choosing-models", "fabrics", "facility-aware-placement", "data", "reference"} +var docOrder = []string{"console", "quick-install", "install", "node-onboarding", "onboarding", + "guardrails", "choosing-models", "fabrics", "facility-aware-placement", "data", "reference", "development"} func docTopics() []docTopic { read := func(path string) func() string {