From 80e0dd252fd920273adcdf11077b36696164d34d Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Tue, 29 Sep 2026 10:14:58 +0200 Subject: [PATCH] Document what changed since v0.9.2, and fabrics for the sites that have them The changelog's unreleased section had stopped at refusing unloadable weights; it now covers the rest: telling independent fabrics apart, their subnet managers and maps, NVLink on every build and across nodes, the multi-node launch fixes and vLLM's new default, doctor, node load, editors and the Responses API, the chat and mirror pages, a person's own use, the admin pages, the new metrics, and how to upgrade. docs/fabrics.md is new, for a site with InfiniBand or NVLink: what llmgw does with them, what each node and fabric needs first, declaring fabrics, checking a node with doctor, what the pages and metrics show and which to alert on, what an engine is given, and where to look when a replica spanning nodes will not run. It says plainly that none of it has yet run on such hardware. install.md, node-onboarding.md, onboarding.md, the README and the example agent configuration no longer say the portable binary cannot see NVLink or that a fabric's name cannot be checked, and describe the account page and connecting an editor. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 166 ++++++++++++++++++++++++- README.md | 17 ++- deploy/examples/agent.yaml | 16 ++- docs/fabrics.md | 177 +++++++++++++++++++++++++++ docs/install.md | 69 +++++------ docs/node-onboarding.md | 18 ++- docs/onboarding.md | 29 ++++- docs/reference.md | 5 +- internal/agent/discovery/discover.go | 5 +- internal/config/example_test.go | 5 +- 10 files changed, 440 insertions(+), 67 deletions(-) create mode 100644 docs/fabrics.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 8496286..278ba2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,170 @@ APIs change between minors without a deprecation period. ## Unreleased +### Independent InfiniBand fabrics, told apart and drawn + +A fabric was whatever `agent.yaml` called it, and nothing checked. At a site +with several independent fabrics, two given one name would have had a replica +spread across both — started, then hung on its first collective. Now: + +- **Identity.** Each active InfiniBand port reports the master subnet manager + it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`), + asked once a minute through one port per fabric. Ports whose managers are + listed together are one physical fabric, so a failover does not split one in + two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent + installs `infiniband-diags` where there are adapters and no `sminfo`. +- **Placement** divides a declared fabric by that identity and forms a replica + on one part only. A node whose adapters for one name are cabled to two + fabrics is refused; a node that cannot say is kept apart from those that can. +- **Subnet managers.** The cluster page lists each fabric's master, its + priority, its standbys and its master changes over the last day, and warns + when a fabric has no standby, when its ports disagree on the master, when a + port is up with no subnet manager (INIT), and when one name covers two + fabrics or two names one fabric. +- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on + each heartbeat: state, physical state, rate, error and data counters. A port + going down takes its fabric off the node within seconds, and back when it + returns — it used to wait for the agent to restart. +- **Maps.** Every five minutes one node on each fabric + runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws + the fabric: switches in their tiers, adapters under their leaves, cables + coloured by rate, degraded ones — trained below the rest of their kind — + marked in red, rail alignment, and the cluster's own adapters live, with + their traffic running along the cables. It zooms, pans, highlights what a + switch or adapter touches, and opens detail on a click. Placement prefers + the nodes of one leaf and puts nodes with a degraded cable last. + +### NVLink, on every build and across nodes + +- **The release binary sees NVLink.** It is built without cgo, so without NVML, + which was the only source of NVLink peers: on every NVLink machine the + portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink + peers, each card's NUMA node, and how near each network adapter is. +- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs + within it, then — for a multi-node shard — nearness to the fabric's adapters, + then NUMA nodes spanned. It used to grow one card's peers, which took cards + joined to that one but not to each other. +- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are + found from each GPU's fabric state, ClusterUUID and CliqueId, with + `nvidia-imex` and its channels. A replica spanning nodes is placed inside one + domain first, on nodes whose every card is joined and whose IMEX is up, and + its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback. + `/admin/nvlink` draws each domain as a rack of trays and says what keeps a + tray out. A node's page shows its GPU-by-GPU NVLink matrix. + +### Multi-node launches fixed before they meet hardware + +- **vLLM launches multi-node its own way by default.** An empty + `engines.vllm.launch_mode` meant `ray`, never checked against a real cluster; + it now means `mp`, every node running `vllm serve` with `--nnodes`, + `--node-rank` and `--master-addr`, the workers `--headless`, as vLLM + documents. The `mp` path used to give workers no rank and the leader's + address as their own. Set `launch_mode: ray` to keep Ray. +- **NCCL bootstraps over the node's own interface.** The agent's default of + `eth0` meant it never looked for the interface holding its management + address, and the control plane sent one name for every node. A node on + `ens3` or `bond0` had NCCL bind nothing. +- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4 + address; NCCL's default is a v1 link-local one that does not route. +- A Ray worker now waits for its head before joining. +- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs. + +### `llmgw doctor` + +Checks a node for what llmgw needs of its GPUs and fabric, and says what is +wrong and how to fix it, most serious first: NVSwitch without the fabric +manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and +`ib_uverbs`, each port's link and subnet manager, a fabric with no standby, +RoCE without a v2 GID, the agent's declared fabrics against the adapters, the +interface NCCL will use, the unit's memlock limit, and the NCCL environment a +shard on each fabric would be given. `--json` writes every fact it read, to +send back; it exits non-zero when something fails. + +### Each node's CPU and memory, and a page per node + +The agent reads `/proc` between heartbeats: CPU across all cores, load, memory +used and available, and each engine's cores and resident memory. The cluster +page's node table shows CPU and RAM as bars; each node has a page with its +current load, its engines, its fabric ports, its GPUs, and an hour of charts +from Prometheus. + +### Connect an editor, and the Responses API + +- The account page makes a key for **Continue**, **Codex** or **Cline** and + shows what the tool needs, filled in: Continue's assistant file, Codex's + provider and profile, Cline's four settings. Continue and Codex get a one-line + command (`curl -fsSL /setup/?model=… | sh`) whose script holds + no secret — the key comes from the environment — keeps a copy of whatever it + replaces, and leaves the rest of the file alone. +- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to + chat completions and back — instructions, messages, function and custom + tools with their calls and outputs, usage, streamed or not — down the same + path as any request. + +### The chat page + +- A model that is not running shows a progress bar while it loads — waiting + for a node, downloading, starting — and the message is sent once it is up, + rather than failing after 45 seconds with "the model is loading". +- A finished reply is rendered as markdown, by the server under the model + cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose. +- Stop, while a reply streams; its tokens, speed and time to first token under + it; a system prompt and a temperature; and the conversation kept in the + browser across a reload — the gateway still keeps none. + +### The mirror page + +- **Explore models** opens on what is trending on Hugging Face, with views for + new and most downloaded, filters for task and size, and each repository's + parameters, architecture (✓ where the nodes' llama.cpp loads it), context, + base model and license. "Only what these nodes can run" is on by default. +- A repository opens beneath its row with its model card, and a download + started there starts in place, its progress followed in the table above. +- What the mirror holds is grouped by model family. +- An importance matrix published beside a model's quantisations is no longer + offered as a model. + +### Your own use + +The account page (was "Your keys") shows the last thirty days of a person's own +use and, for each place they spend, a bar for every budget that applies — their +own, or a project's shared with others, said as such. + +### The admin pages + +- The cluster page leads with capacity, nodes, models and replicas, then usage; + it refreshes in place instead of reloading, and its usage table names people + and projects instead of ids. +- The usage page's legend gives each line's total, not its last hour. +- The people page keeps an account on one line and its rarer actions in a menu. +- Every page fits a phone: a wide table scrolls in its own box. +- Static files are named by their content, so a deploy's new styles and + scripts reach browsers at once instead of a day later. + +### Metrics + +`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`, +`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`, +`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`, +`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`, +`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`, +`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`. + +### Fixes + +- PostgreSQL queries with an apostrophe in a comment failed: the placeholder + rewriting read it as the start of a string. Comments are now skipped. +- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU. + +### Upgrading + +Upgrade the control plane first, then the agents: an older agent is served as +before and simply reports none of the new measurements. Existing agent units +keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node +that set `mgmt_interface` keeps it; one that did not now uses the interface +holding its management address rather than `eth0`. Multi-node vLLM replicas +start with `mp` unless `launch_mode: ray` is set. + ### An organization's own page A member of an organization can be made its **manager** (`users join --org … @@ -98,7 +262,7 @@ before release. `llmgw users move` is gone: use `join` and `leave`. Each page drew its own menu, and no two agreed: different links, in a different order, the page you were on sometimes listed and sometimes not. Every signed-in page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the -left for an administrator, Chat, Your keys and Sign out on the right for +left for an administrator, Chat, Account and Sign out on the right for everyone — with the current page marked. The chat and key pages name the person's project by its path rather than its id. diff --git a/README.md b/README.md index 2df8bcb..108519b 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # llmgw -A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and -heterogeneous GPUs. +A single-binary LLM gateway for clusters with multiple InfiniBand fabrics, +NVLink domains and heterogeneous GPUs. It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md). @@ -49,15 +49,21 @@ a header would let any client promote itself into the top priority band. One thing is worth saying plainly: multi-node replicas are proven on a simulated cluster and on real hardware -without a fabric. The InfiniBand, NVML and NCCL paths are written against the -formats and failure modes those tools actually have, but they have not yet run -on a machine that has them. +without a fabric. The InfiniBand, NVLink and NCCL paths — telling fabrics apart +by their subnet managers, mapping them, finding NVLink domains, launching across +nodes — are written against the formats and failure modes those tools actually +have and tested against them, but they have not yet run on a machine that has +them. On such a machine, run `llmgw doctor --json` first and keep the report; +[docs/fabrics.md](docs/fabrics.md) says what to look at after that. ## Documentation - [docs/install.md](docs/install.md) — installing from nothing: control plane, nodes, engines, weights, first request - [docs/quickstart.md](docs/quickstart.md) — the short path +- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what + llmgw needs of them, checking a node with `llmgw doctor`, and what to look at + when a replica spanning nodes will not run - [docs/choosing-models.md](docs/choosing-models.md) — which models to run, how many of each, and where the weights should live - [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and @@ -91,6 +97,7 @@ llmgw ctl replicas --model llama # replicas and their shards llmgw ctl explain-placement llama # why a model has not been placed llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs +sudo llmgw doctor # on a node: what its GPUs, NVLink and fabric need ``` `ctl` talks to the admin port (`:9090` by default), not the client-facing one. diff --git a/deploy/examples/agent.yaml b/deploy/examples/agent.yaml index 2b46506..c032ab7 100644 --- a/deploy/examples/agent.yaml +++ b/deploy/examples/agent.yaml @@ -47,26 +47,24 @@ heartbeat_interval: 5s reconnect_base: 1s reconnect_max: 30s -# Fabric labelling is explicit and there is no way around it. OpenSM's default -# subnet prefix is fe80:: on every fabric, so a matching prefix proves nothing -# and a wrong guess produces a replica that starts, passes health checks and -# then hangs on its first all-reduce (plan §5.2). +# Each fabric this node's adapters are on, by name. The name is checked against +# what the adapters report — the subnet managers every port on one fabric sees — +# so two independent fabrics given one name are placed as two and the cluster +# page says so; give each its own. The rate is measured, not read from here. # # A node with no fabrics is its own single-node island: it can host single-node -# replicas and will never be given a multi-node one. +# replicas and will never be given a multi-node one. docs/fabrics.md has more. fabrics: - name: ib-a hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3] - rate_gbps: 200 discovery: # Worth setting where the driver lives outside the default paths, which # containerized driver installs often do. nvidia_smi: "" ib_root: /sys/class/infiniband - # NVML is the only source of NVLink topology, which is what keeps a - # tensor-parallel group off the PCIe path inside a node. It needs a binary - # built with `-tags nvml`; otherwise this silently falls back to nvidia-smi. + # NVLink topology comes from `nvidia-smi topo -m` on every build; a binary + # built with `-tags nvml` reads it from NVML instead, when this is set. prefer_nvml: true # Where this node keeps model weights besides /var/lib/llmgw-weights, which is diff --git a/docs/fabrics.md b/docs/fabrics.md new file mode 100644 index 0000000..a1ea392 --- /dev/null +++ b/docs/fabrics.md @@ -0,0 +1,177 @@ +# InfiniBand fabrics and NVLink + +For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both +— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what +llmgw does with them, what it needs from you before it can, how to check a node +in one command, and where to look when a replica spanning nodes will not run. + +Read [install.md](install.md) first; this is the part of it that only matters +once there is a fabric. + +> **Where this stands.** Every piece here was written against the real formats +> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and +> `nvidia-smi -q`, and tested against them — but none of it has yet run on a +> site with the hardware. Run `llmgw doctor --json` on two or three nodes first +> and keep the reports: if a parser meets a format it does not know, that report +> is what fixes it, and the fix is a control plane upgrade, not one per node. + +## What llmgw does with them + +**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out). +NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a +rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes +is placed, in order of preference: + +1. inside one **multi-node NVLink domain**, if any has enough usable nodes: + tensor parallelism between its nodes then runs over NVLink; +2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one + leaf switch and putting nodes with a degraded cable last. + +It is never split across two fabrics or two NVLink domains. Within a node, the +GPUs a shard gets are the set with the most NVLink pairs, then the nearest to +the fabric's adapters, then on one NUMA node. + +**A fabric has a name and an identity.** The name is what `agent.yaml` says; the +identity is what the adapters report — on InfiniBand the subnet managers every +port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name +is how you address a fabric; the identity is how llmgw catches the name being +wrong: two independent fabrics given one name are placed as two, and the +cluster page says so. + +## Before a node joins + +On every GPU node: + +| What | Why | Check | +|---|---|---| +| NVIDIA driver | everything | `nvidia-smi` | +| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` | +| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` | +| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` | +| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` | + +On an NVL72 or NVLink Switch system, also: + +| What | Why | Check | +|---|---|---| +| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` | +| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` | +| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` | + +On each InfiniBand fabric: + +- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and + stay there; without a standby, the fabric keeps its routes if the master + stops but cannot recover from any change after. llmgw watches this but does + not run one. +- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a + distinct `subnet_prefix` in each subnet manager's configuration, which makes + GIDs unique across fabrics; llmgw does not need it. + +## Declaring the fabrics + +In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name: + +```yaml +fabrics: + - name: ib-compute + hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9] + - name: ib-storage + hcas: [mlx5_4] +``` + +or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)). +An adapter belongs to one fabric; a node may be on several. A replica on +`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`). + +The rate is measured, not taken from the file: a fabric's rate on a node is the +slowest of its active adapters, because a collective runs at the speed of its +slowest link. + +The agent checks the declaration against the adapters on every heartbeat. An +adapter that is missing, or whose port is down, takes that fabric off the node +within seconds — the node stops being offered multi-node work on it — and +returns it when the port comes back. + +## Checking a node: `llmgw doctor` + +``` +sudo llmgw doctor # uses /etc/llmgw/agent.yaml when present +sudo llmgw doctor --json > $(hostname)-doctor.json +``` + +It reads the node and says what is wrong and how to fix it, most serious first, +and exits non-zero when anything fails: + +``` +FAIL nvlink this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs + → systemctl enable --now nvidia-fabricmanager (its version must match the driver's) +WARN fabric nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory + → modprobe nvidia_peermem +OK gpu 8 GPUs: 8× NVIDIA H100 80GB HBM3 +OK nvlink NVLink groups: [0 1 2 3 4 5 6 7] +OK fabric mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER) +OK fabric subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14) +OK agent fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s +OK agent NCCL bootstraps over ens3 (10.1.0.21) +``` + +followed by each GPU's NVLink peers and nearest adapter, each port's state, +rate and errors, and the NCCL environment a shard on each fabric would be given. +`--json` writes every fact it read as well as what it concluded — send that +file when the report is not enough. + +## Watching them + +- **Cluster page, Fabrics.** Each declared fabric and what it physically is; + a table of subnet managers per fabric — master, priority, standbys, the last + change of master — and every disagreement: one name on two fabrics, two names + on one, an adapter up but undeclared, ports up with no subnet manager, a + fabric with no standby, a recent failover. +- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of + its nodes every five minutes: switches in their tiers, adapters under their + leaves, cables coloured by rate, degraded ones in red, rail alignment, and + the cluster's own adapters live with their traffic. Click anything for its + detail. +- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which + GPUs are joined, which node's IMEX is down, which link runs degraded. +- **A node's page.** Its ports with state, rate, errors and what they are + physically on; its GPU-by-GPU NVLink matrix; its load. +- **Metrics**, for alerts: + +| Metric | Alert when | +|---|---| +| `llmgw_ib_port_active` | a declared port is 0 | +| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising | +| `llmgw_ib_port_rate_gbps` | below its usual rate | +| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 | +| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) | +| `llmgw_fabric_problems{level="fail"}` | above 0 | +| `llmgw_gpu_nvlink_joined` | 0 | +| `llmgw_imex_up` | 0 | + +## What an engine is given + +For a shard on an InfiniBand or RoCE fabric: + +| Variable | Value | +|---|---| +| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others | +| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) | +| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address | +| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address | +| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain | + +A profile's `env` overrides any of them. vLLM launches a multi-node replica +with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default): +every node runs `vllm serve` with its rank and the leader's address, the +workers headless. `launch_mode: ray` is there for a vLLM too old for that. + +## When a replica spanning nodes will not run + +| It | Look at | +|---|---| +| is not placed | `llmgw ctl explain-placement `: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics | +| starts and hangs on its first collective | the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID | +| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe | +| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports | diff --git a/docs/install.md b/docs/install.md index 92147f0..5d42ce0 100644 --- a/docs/install.md +++ b/docs/install.md @@ -318,32 +318,26 @@ hardware to it. ### NVLink, if your GPUs have it -The published binary cannot see NVLink topology. NVML is the only source of it, -its bindings need cgo, and the portable binary is built without cgo on purpose -so it runs on a node whose libc is older than the builder's. +The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink +joins, each card's NUMA node, and which network adapter is nearest each. A +tensor-parallel group is then the set of cards with the most NVLink between +them — on a PCIe box where bridges pair specific cards, a group straddling two +bridges falls back to PCIe and runs several times slower, and nothing else +would report it. -Without that topology, a tensor-parallel group takes the lowest free GPU -indices. On an SXM board with NVSwitch that is correct, because every card is -equidistant. On a PCIe box where NVLink bridges pair specific cards, a group -straddling two bridges falls back to PCIe and runs several times slower — and -nothing reports it, because from the outside it is simply a replica that is -slow. - -For those nodes, install the `-nvml` build published beside the portable one: +On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the +driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an +NVLink Switch system, the GPUs of several nodes form one domain, and a replica +can run tensor parallelism across its nodes — which needs `nvidia-imex` and an +IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor` +checks it: ``` -sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw +sudo llmgw doctor ``` -It is dynamically linked, so it needs a libc no older than the builder's and the -NVIDIA driver present. Check it took effect with: - -``` -llmgw ctl nodes --json | grep -c nvlink_peers # 0 means the topology is unknown -``` - -`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a -flat zero everywhere means the topology is unknown rather than absent. +A `-nvml` build is still published beside the portable one; it reads NVLink +from NVML directly, and needs a libc no older than the builder's. ### InfiniBand, if you have it @@ -353,17 +347,17 @@ ibstat # must list your HCAs with State: Active ls /sys/class/infiniband ``` -The agent checks what you declare against what the node has. An adapter that is -not there, or that has no ACTIVE port — physically linked but with no subnet -manager, which survives a reboot and looks like a software problem — takes that -fabric out of service on this node, loudly, in the log. Without that check the -replica starts, passes its health checks, and hangs on its first all-reduce. +Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem` +should be loaded for GPUDirect RDMA. You name each fabric when you onboard the +node, and llmgw checks the name against what the adapters report: every port on +one fabric sees the same subnet managers, so two independent fabrics given one +name are placed as two, and the cluster page says so. An adapter that is not +there, or whose port goes down, takes that fabric off the node within seconds, +and gives it back when the port returns. -Fabric membership is configuration, not discovery, and there is no way around -that: OpenSM's default subnet prefix is the same on every fabric, so a matching -prefix proves nothing and a wrong guess produces a replica that starts, passes -health checks, and then hangs on its first all-reduce. You name the fabric when -you onboard the node, and two nodes are only in the same fabric if you said so. +[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and +fabric pages show, and what to look at when a replica spanning nodes will not +run. Run `llmgw doctor` on the node before it joins. ### An engine @@ -1048,6 +1042,8 @@ requests. On a slow model, set `timeouts.generation` higher. and what belongs on which port - `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what each one assumes about this system +- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them, + checking a node, and what to look at when a replica spanning nodes will not run - `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding deliberately does not do - `docs/onboarding.md` — giving people accounts and keys @@ -1069,8 +1065,9 @@ requests. On a slow model, set `timeouts.generation` higher. agent failover work and there is a pattern in `deploy/ha/`, but every cluster this has been run on in anger has had one. A single control plane is the well-trodden path. -- **Multi-node replicas have not been run on real InfiniBand.** The scheduling, - the fabric invariant and the rendezvous are exercised against simulated - hardware only. Single-node placement on real GPUs is the well-trodden path; - if you are the first to run a TP group across real cards, expect to find - something. +- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The + scheduling, the fabric identity, the maps and the launch are written against + the real tools' formats and tested against them, not yet run on hardware that + has them. Single-node placement on real GPUs is the well-trodden path; if you + are the first to run a TP group across real cards, run `llmgw doctor --json` + on each node first and keep the reports. diff --git a/docs/node-onboarding.md b/docs/node-onboarding.md index a0ec78e..9a3fe39 100644 --- a/docs/node-onboarding.md +++ b/docs/node-onboarding.md @@ -178,18 +178,24 @@ was cut short. ## Fabrics -`--fabric` is how a node says which InfiniBand fabric it is attached to, and -there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on -every fabric, so a matching prefix proves nothing and a wrong guess produces a -replica that starts, passes health checks, and then hangs on its first -all-reduce. +`--fabric` is how a node says which InfiniBand fabric it is attached to. The +name is checked against what the adapters report — every port on one fabric +sees the same subnet managers — so two independent fabrics given one name are +placed as two, and the cluster page says so; give each fabric its own name. +The agent installs `infiniband-diags`, which that check needs, if it is missing. + +Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the +subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines, +and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the +rest. Omit it for a node with no fabric. Such a node is its own island: it can host single-node replicas and will never be given a multi-node one. Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them explicitly. An adapter must never appear under two fabrics — the agent refuses -that configuration, because the consequence is the hang above. +that configuration: a replica given adapters on two fabrics starts, passes its +health checks, and hangs on its first all-reduce. ## Where the driver is diff --git a/docs/onboarding.md b/docs/onboarding.md index 51e2c0e..91cb1f5 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -47,9 +47,32 @@ llmgw users approve ayse@example.com --project Acme/ml/training --store … # ``` That is the whole operator loop. From then on they sign in at `/login`, and -`/profile` is where they create keys, see when each was last used, and revoke the -ones they are done with. A key is shown once on the page that creates it and -never again. +their account page (`/profile`) is where they create keys, see when each was +last used, revoke the ones they are done with, and see their own use over the +last thirty days against every budget that applies to them. A key is shown once +on the page that creates it and never again. + +### Connecting an editor + +The account page's **Connect your editor** makes a key for Continue, Codex or +Cline and shows what the tool needs with that key, this gateway's address and a +model already in it: + +- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`, + so their `config.yaml` is not touched. +- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with + `codex --profile llmgw`; their default model stays as it was. Codex speaks the + Responses API, which the gateway translates at `/v1/responses`. +- **Cline** — four settings to paste into its own screen, since it keeps them + in VS Code's secret storage where no file reaches. + +For Continue and Codex there is a one-line command, +`curl -fsSL /setup/?model=… | sh`. Its script is served without +signing in and holds no secret — the key comes from the environment — and it +keeps a copy of anything it replaces. Revoking the key disconnects the tool. + +How well an editor's agent mode works depends on the model: it needs one that +calls tools well, which the coder models do and small general ones often do not. **This gateway sends no email.** There is no "your account was approved" message and no password reset, so tell people what to expect: they sign up, they tell diff --git a/docs/reference.md b/docs/reference.md index 292646e..910ef09 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -793,8 +793,9 @@ IBRoot is the sysfs InfiniBand directory. `bool` -PreferNVML uses the NVML backend when this binary has one. It is the only -source of NVLink topology. +PreferNVML uses the NVML backend when this binary has one. NVLink +topology is read from nvidia-smi topo -m on every build; NVML, where it +is used, is the more exact source of the peers. ### `skip_gpus` diff --git a/internal/agent/discovery/discover.go b/internal/agent/discovery/discover.go index d714b7b..e8081b7 100644 --- a/internal/agent/discovery/discover.go +++ b/internal/agent/discovery/discover.go @@ -21,8 +21,9 @@ type Options struct { NvidiaSMI string `koanf:"nvidia_smi"` // IBRoot is the sysfs InfiniBand directory. IBRoot string `koanf:"ib_root"` - // PreferNVML uses the NVML backend when this binary has one. It is the only - // source of NVLink topology. + // PreferNVML uses the NVML backend when this binary has one. NVLink + // topology is read from nvidia-smi topo -m on every build; NVML, where it + // is used, is the more exact source of the peers. PreferNVML bool `koanf:"prefer_nvml"` // SkipGPUs omits GPU discovery. A node without nvidia-smi is already // found to have none, so this is for one whose driver is broken. diff --git a/internal/config/example_test.go b/internal/config/example_test.go index 26ed1b0..744c37a 100644 --- a/internal/config/example_test.go +++ b/internal/config/example_test.go @@ -36,9 +36,8 @@ func TestTheDocumentedAgentExampleLoads(t *testing.T) { if len(f.HCAs) == 0 { t.Error("the fabric's adapters did not bind") } - if f.RateGbps == 0 { - t.Error("the fabric's rate did not bind") - } + // The rate is not documented: the agent measures it, as the slowest of + // the fabric's active ports, and a figure written here would be ignored. // node_id is left to the hostname, so the example need not say it; where // the control plane is, nothing can find out. if cfg.Server == "" {