Document what changed since v0.9.2, and fabrics for the sites that have them
The changelog's unreleased section had stopped at refusing unloadable weights; it now covers the rest: telling independent fabrics apart, their subnet managers and maps, NVLink on every build and across nodes, the multi-node launch fixes and vLLM's new default, doctor, node load, editors and the Responses API, the chat and mirror pages, a person's own use, the admin pages, the new metrics, and how to upgrade. docs/fabrics.md is new, for a site with InfiniBand or NVLink: what llmgw does with them, what each node and fabric needs first, declaring fabrics, checking a node with doctor, what the pages and metrics show and which to alert on, what an engine is given, and where to look when a replica spanning nodes will not run. It says plainly that none of it has yet run on such hardware. install.md, node-onboarding.md, onboarding.md, the README and the example agent configuration no longer say the portable binary cannot see NVLink or that a fabric's name cannot be checked, and describe the account page and connecting an editor. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed
1 parent
f989e67e1a
commit
80e0dd252f
10 files changed
+440
-67
No files matched your search
+165
-1
@@ -21,6 +21,170 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Independent InfiniBand fabrics, told apart and drawn
|
||||
|
||||
A fabric was whatever `agent.yaml` called it, and nothing checked. At a site
|
||||
with several independent fabrics, two given one name would have had a replica
|
||||
spread across both — started, then hung on its first collective. Now:
|
||||
|
||||
- **Identity.** Each active InfiniBand port reports the master subnet manager
|
||||
it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`),
|
||||
asked once a minute through one port per fabric. Ports whose managers are
|
||||
listed together are one physical fabric, so a failover does not split one in
|
||||
two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent
|
||||
installs `infiniband-diags` where there are adapters and no `sminfo`.
|
||||
- **Placement** divides a declared fabric by that identity and forms a replica
|
||||
on one part only. A node whose adapters for one name are cabled to two
|
||||
fabrics is refused; a node that cannot say is kept apart from those that can.
|
||||
- **Subnet managers.** The cluster page lists each fabric's master, its
|
||||
priority, its standbys and its master changes over the last day, and warns
|
||||
when a fabric has no standby, when its ports disagree on the master, when a
|
||||
port is up with no subnet manager (INIT), and when one name covers two
|
||||
fabrics or two names one fabric.
|
||||
- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on
|
||||
each heartbeat: state, physical state, rate, error and data counters. A port
|
||||
going down takes its fabric off the node within seconds, and back when it
|
||||
returns — it used to wait for the agent to restart.
|
||||
- **Maps.** Every five minutes one node on each fabric
|
||||
runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws
|
||||
the fabric: switches in their tiers, adapters under their leaves, cables
|
||||
coloured by rate, degraded ones — trained below the rest of their kind —
|
||||
marked in red, rail alignment, and the cluster's own adapters live, with
|
||||
their traffic running along the cables. It zooms, pans, highlights what a
|
||||
switch or adapter touches, and opens detail on a click. Placement prefers
|
||||
the nodes of one leaf and puts nodes with a degraded cable last.
|
||||
|
||||
### NVLink, on every build and across nodes
|
||||
|
||||
- **The release binary sees NVLink.** It is built without cgo, so without NVML,
|
||||
which was the only source of NVLink peers: on every NVLink machine the
|
||||
portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink
|
||||
peers, each card's NUMA node, and how near each network adapter is.
|
||||
- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs
|
||||
within it, then — for a multi-node shard — nearness to the fabric's adapters,
|
||||
then NUMA nodes spanned. It used to grow one card's peers, which took cards
|
||||
joined to that one but not to each other.
|
||||
- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are
|
||||
found from each GPU's fabric state, ClusterUUID and CliqueId, with
|
||||
`nvidia-imex` and its channels. A replica spanning nodes is placed inside one
|
||||
domain first, on nodes whose every card is joined and whose IMEX is up, and
|
||||
its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback.
|
||||
`/admin/nvlink` draws each domain as a rack of trays and says what keeps a
|
||||
tray out. A node's page shows its GPU-by-GPU NVLink matrix.
|
||||
|
||||
### Multi-node launches fixed before they meet hardware
|
||||
|
||||
- **vLLM launches multi-node its own way by default.** An empty
|
||||
`engines.vllm.launch_mode` meant `ray`, never checked against a real cluster;
|
||||
it now means `mp`, every node running `vllm serve` with `--nnodes`,
|
||||
`--node-rank` and `--master-addr`, the workers `--headless`, as vLLM
|
||||
documents. The `mp` path used to give workers no rank and the leader's
|
||||
address as their own. Set `launch_mode: ray` to keep Ray.
|
||||
- **NCCL bootstraps over the node's own interface.** The agent's default of
|
||||
`eth0` meant it never looked for the interface holding its management
|
||||
address, and the control plane sent one name for every node. A node on
|
||||
`ens3` or `bond0` had NCCL bind nothing.
|
||||
- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4
|
||||
address; NCCL's default is a v1 link-local one that does not route.
|
||||
- A Ray worker now waits for its head before joining.
|
||||
- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs.
|
||||
|
||||
### `llmgw doctor`
|
||||
|
||||
Checks a node for what llmgw needs of its GPUs and fabric, and says what is
|
||||
wrong and how to fix it, most serious first: NVSwitch without the fabric
|
||||
manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and
|
||||
`ib_uverbs`, each port's link and subnet manager, a fabric with no standby,
|
||||
RoCE without a v2 GID, the agent's declared fabrics against the adapters, the
|
||||
interface NCCL will use, the unit's memlock limit, and the NCCL environment a
|
||||
shard on each fabric would be given. `--json` writes every fact it read, to
|
||||
send back; it exits non-zero when something fails.
|
||||
|
||||
### Each node's CPU and memory, and a page per node
|
||||
|
||||
The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
|
||||
used and available, and each engine's cores and resident memory. The cluster
|
||||
page's node table shows CPU and RAM as bars; each node has a page with its
|
||||
current load, its engines, its fabric ports, its GPUs, and an hour of charts
|
||||
from Prometheus.
|
||||
|
||||
### Connect an editor, and the Responses API
|
||||
|
||||
- The account page makes a key for **Continue**, **Codex** or **Cline** and
|
||||
shows what the tool needs, filled in: Continue's assistant file, Codex's
|
||||
provider and profile, Cline's four settings. Continue and Codex get a one-line
|
||||
command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
|
||||
no secret — the key comes from the environment — keeps a copy of whatever it
|
||||
replaces, and leaves the rest of the file alone.
|
||||
- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
|
||||
chat completions and back — instructions, messages, function and custom
|
||||
tools with their calls and outputs, usage, streamed or not — down the same
|
||||
path as any request.
|
||||
|
||||
### The chat page
|
||||
|
||||
- A model that is not running shows a progress bar while it loads — waiting
|
||||
for a node, downloading, starting — and the message is sent once it is up,
|
||||
rather than failing after 45 seconds with "the model is loading".
|
||||
- A finished reply is rendered as markdown, by the server under the model
|
||||
cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
|
||||
- Stop, while a reply streams; its tokens, speed and time to first token under
|
||||
it; a system prompt and a temperature; and the conversation kept in the
|
||||
browser across a reload — the gateway still keeps none.
|
||||
|
||||
### The mirror page
|
||||
|
||||
- **Explore models** opens on what is trending on Hugging Face, with views for
|
||||
new and most downloaded, filters for task and size, and each repository's
|
||||
parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
|
||||
base model and license. "Only what these nodes can run" is on by default.
|
||||
- A repository opens beneath its row with its model card, and a download
|
||||
started there starts in place, its progress followed in the table above.
|
||||
- What the mirror holds is grouped by model family.
|
||||
- An importance matrix published beside a model's quantisations is no longer
|
||||
offered as a model.
|
||||
|
||||
### Your own use
|
||||
|
||||
The account page (was "Your keys") shows the last thirty days of a person's own
|
||||
use and, for each place they spend, a bar for every budget that applies — their
|
||||
own, or a project's shared with others, said as such.
|
||||
|
||||
### The admin pages
|
||||
|
||||
- The cluster page leads with capacity, nodes, models and replicas, then usage;
|
||||
it refreshes in place instead of reloading, and its usage table names people
|
||||
and projects instead of ids.
|
||||
- The usage page's legend gives each line's total, not its last hour.
|
||||
- The people page keeps an account on one line and its rarer actions in a menu.
|
||||
- Every page fits a phone: a wide table scrolls in its own box.
|
||||
- Static files are named by their content, so a deploy's new styles and
|
||||
scripts reach browsers at once instead of a day later.
|
||||
|
||||
### Metrics
|
||||
|
||||
`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
|
||||
`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
|
||||
`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
|
||||
`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
|
||||
`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
|
||||
`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
|
||||
|
||||
### Fixes
|
||||
|
||||
- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
|
||||
rewriting read it as the start of a string. Comments are now skipped.
|
||||
- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
|
||||
|
||||
### Upgrading
|
||||
|
||||
Upgrade the control plane first, then the agents: an older agent is served as
|
||||
before and simply reports none of the new measurements. Existing agent units
|
||||
keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
|
||||
that set `mgmt_interface` keeps it; one that did not now uses the interface
|
||||
holding its management address rather than `eth0`. Multi-node vLLM replicas
|
||||
start with `mp` unless `launch_mode: ray` is set.
|
||||
|
||||
### An organization's own page
|
||||
|
||||
A member of an organization can be made its **manager** (`users join --org …
|
||||
@@ -98,7 +262,7 @@ before release. `llmgw users move` is gone: use `join` and `leave`.
|
||||
Each page drew its own menu, and no two agreed: different links, in a different
|
||||
order, the page you were on sometimes listed and sometimes not. Every signed-in
|
||||
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
|
||||
left for an administrator, Chat, Your keys and Sign out on the right for
|
||||
left for an administrator, Chat, Account and Sign out on the right for
|
||||
everyone — with the current page marked. The chat and key pages name the
|
||||
person's project by its path rather than its id.
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# llmgw
|
||||
|
||||
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
|
||||
heterogeneous GPUs.
|
||||
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
|
||||
NVLink domains and heterogeneous GPUs.
|
||||
|
||||
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
||||
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
||||
@@ -49,15 +49,21 @@ a header would let any client promote itself into the top priority band.
|
||||
|
||||
One thing is worth saying plainly:
|
||||
multi-node replicas are proven on a simulated cluster and on real hardware
|
||||
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
|
||||
formats and failure modes those tools actually have, but they have not yet run
|
||||
on a machine that has them.
|
||||
without a fabric. The InfiniBand, NVLink and NCCL paths — telling fabrics apart
|
||||
by their subnet managers, mapping them, finding NVLink domains, launching across
|
||||
nodes — are written against the formats and failure modes those tools actually
|
||||
have and tested against them, but they have not yet run on a machine that has
|
||||
them. On such a machine, run `llmgw doctor --json` first and keep the report;
|
||||
[docs/fabrics.md](docs/fabrics.md) says what to look at after that.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [docs/install.md](docs/install.md) — installing from nothing: control plane,
|
||||
nodes, engines, weights, first request
|
||||
- [docs/quickstart.md](docs/quickstart.md) — the short path
|
||||
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
||||
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
||||
when a replica spanning nodes will not run
|
||||
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
|
||||
how many of each, and where the weights should live
|
||||
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
|
||||
@@ -91,6 +97,7 @@ llmgw ctl replicas --model llama # replicas and their shards
|
||||
llmgw ctl explain-placement llama # why a model has not been placed
|
||||
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
|
||||
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
|
||||
sudo llmgw doctor # on a node: what its GPUs, NVLink and fabric need
|
||||
```
|
||||
|
||||
`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
|
||||
|
||||
@@ -47,26 +47,24 @@ heartbeat_interval: 5s
|
||||
reconnect_base: 1s
|
||||
reconnect_max: 30s
|
||||
|
||||
# Fabric labelling is explicit and there is no way around it. OpenSM's default
|
||||
# subnet prefix is fe80:: on every fabric, so a matching prefix proves nothing
|
||||
# and a wrong guess produces a replica that starts, passes health checks and
|
||||
# then hangs on its first all-reduce (plan §5.2).
|
||||
# Each fabric this node's adapters are on, by name. The name is checked against
|
||||
# what the adapters report — the subnet managers every port on one fabric sees —
|
||||
# so two independent fabrics given one name are placed as two and the cluster
|
||||
# page says so; give each its own. The rate is measured, not read from here.
|
||||
#
|
||||
# A node with no fabrics is its own single-node island: it can host single-node
|
||||
# replicas and will never be given a multi-node one.
|
||||
# replicas and will never be given a multi-node one. docs/fabrics.md has more.
|
||||
fabrics:
|
||||
- name: ib-a
|
||||
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
|
||||
rate_gbps: 200
|
||||
|
||||
discovery:
|
||||
# Worth setting where the driver lives outside the default paths, which
|
||||
# containerized driver installs often do.
|
||||
nvidia_smi: ""
|
||||
ib_root: /sys/class/infiniband
|
||||
# NVML is the only source of NVLink topology, which is what keeps a
|
||||
# tensor-parallel group off the PCIe path inside a node. It needs a binary
|
||||
# built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
|
||||
# NVLink topology comes from `nvidia-smi topo -m` on every build; a binary
|
||||
# built with `-tags nvml` reads it from NVML instead, when this is set.
|
||||
prefer_nvml: true
|
||||
|
||||
# Where this node keeps model weights besides /var/lib/llmgw-weights, which is
|
||||
|
||||
+177
@@ -0,0 +1,177 @@
|
||||
# InfiniBand fabrics and NVLink
|
||||
|
||||
For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both
|
||||
— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what
|
||||
llmgw does with them, what it needs from you before it can, how to check a node
|
||||
in one command, and where to look when a replica spanning nodes will not run.
|
||||
|
||||
Read [install.md](install.md) first; this is the part of it that only matters
|
||||
once there is a fabric.
|
||||
|
||||
> **Where this stands.** Every piece here was written against the real formats
|
||||
> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and
|
||||
> `nvidia-smi -q`, and tested against them — but none of it has yet run on a
|
||||
> site with the hardware. Run `llmgw doctor --json` on two or three nodes first
|
||||
> and keep the reports: if a parser meets a format it does not know, that report
|
||||
> is what fixes it, and the fix is a control plane upgrade, not one per node.
|
||||
|
||||
## What llmgw does with them
|
||||
|
||||
**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out).
|
||||
NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a
|
||||
rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes
|
||||
is placed, in order of preference:
|
||||
|
||||
1. inside one **multi-node NVLink domain**, if any has enough usable nodes:
|
||||
tensor parallelism between its nodes then runs over NVLink;
|
||||
2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one
|
||||
leaf switch and putting nodes with a degraded cable last.
|
||||
|
||||
It is never split across two fabrics or two NVLink domains. Within a node, the
|
||||
GPUs a shard gets are the set with the most NVLink pairs, then the nearest to
|
||||
the fabric's adapters, then on one NUMA node.
|
||||
|
||||
**A fabric has a name and an identity.** The name is what `agent.yaml` says; the
|
||||
identity is what the adapters report — on InfiniBand the subnet managers every
|
||||
port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name
|
||||
is how you address a fabric; the identity is how llmgw catches the name being
|
||||
wrong: two independent fabrics given one name are placed as two, and the
|
||||
cluster page says so.
|
||||
|
||||
## Before a node joins
|
||||
|
||||
On every GPU node:
|
||||
|
||||
| What | Why | Check |
|
||||
|---|---|---|
|
||||
| NVIDIA driver | everything | `nvidia-smi` |
|
||||
| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` |
|
||||
| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` |
|
||||
| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` |
|
||||
| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` |
|
||||
|
||||
On an NVL72 or NVLink Switch system, also:
|
||||
|
||||
| What | Why | Check |
|
||||
|---|---|---|
|
||||
| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` |
|
||||
| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` |
|
||||
| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` |
|
||||
|
||||
On each InfiniBand fabric:
|
||||
|
||||
- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and
|
||||
stay there; without a standby, the fabric keeps its routes if the master
|
||||
stops but cannot recover from any change after. llmgw watches this but does
|
||||
not run one.
|
||||
- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a
|
||||
distinct `subnet_prefix` in each subnet manager's configuration, which makes
|
||||
GIDs unique across fabrics; llmgw does not need it.
|
||||
|
||||
## Declaring the fabrics
|
||||
|
||||
In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name:
|
||||
|
||||
```yaml
|
||||
fabrics:
|
||||
- name: ib-compute
|
||||
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
|
||||
- name: ib-storage
|
||||
hcas: [mlx5_4]
|
||||
```
|
||||
|
||||
or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)).
|
||||
An adapter belongs to one fabric; a node may be on several. A replica on
|
||||
`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`).
|
||||
|
||||
The rate is measured, not taken from the file: a fabric's rate on a node is the
|
||||
slowest of its active adapters, because a collective runs at the speed of its
|
||||
slowest link.
|
||||
|
||||
The agent checks the declaration against the adapters on every heartbeat. An
|
||||
adapter that is missing, or whose port is down, takes that fabric off the node
|
||||
within seconds — the node stops being offered multi-node work on it — and
|
||||
returns it when the port comes back.
|
||||
|
||||
## Checking a node: `llmgw doctor`
|
||||
|
||||
```
|
||||
sudo llmgw doctor # uses /etc/llmgw/agent.yaml when present
|
||||
sudo llmgw doctor --json > $(hostname)-doctor.json
|
||||
```
|
||||
|
||||
It reads the node and says what is wrong and how to fix it, most serious first,
|
||||
and exits non-zero when anything fails:
|
||||
|
||||
```
|
||||
FAIL nvlink this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
|
||||
→ systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
|
||||
WARN fabric nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
|
||||
→ modprobe nvidia_peermem
|
||||
OK gpu 8 GPUs: 8× NVIDIA H100 80GB HBM3
|
||||
OK nvlink NVLink groups: [0 1 2 3 4 5 6 7]
|
||||
OK fabric mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
|
||||
OK fabric subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
|
||||
OK agent fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
|
||||
OK agent NCCL bootstraps over ens3 (10.1.0.21)
|
||||
```
|
||||
|
||||
followed by each GPU's NVLink peers and nearest adapter, each port's state,
|
||||
rate and errors, and the NCCL environment a shard on each fabric would be given.
|
||||
`--json` writes every fact it read as well as what it concluded — send that
|
||||
file when the report is not enough.
|
||||
|
||||
## Watching them
|
||||
|
||||
- **Cluster page, Fabrics.** Each declared fabric and what it physically is;
|
||||
a table of subnet managers per fabric — master, priority, standbys, the last
|
||||
change of master — and every disagreement: one name on two fabrics, two names
|
||||
on one, an adapter up but undeclared, ports up with no subnet manager, a
|
||||
fabric with no standby, a recent failover.
|
||||
- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of
|
||||
its nodes every five minutes: switches in their tiers, adapters under their
|
||||
leaves, cables coloured by rate, degraded ones in red, rail alignment, and
|
||||
the cluster's own adapters live with their traffic. Click anything for its
|
||||
detail.
|
||||
- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which
|
||||
GPUs are joined, which node's IMEX is down, which link runs degraded.
|
||||
- **A node's page.** Its ports with state, rate, errors and what they are
|
||||
physically on; its GPU-by-GPU NVLink matrix; its load.
|
||||
- **Metrics**, for alerts:
|
||||
|
||||
| Metric | Alert when |
|
||||
|---|---|
|
||||
| `llmgw_ib_port_active` | a declared port is 0 |
|
||||
| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising |
|
||||
| `llmgw_ib_port_rate_gbps` | below its usual rate |
|
||||
| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 |
|
||||
| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) |
|
||||
| `llmgw_fabric_problems{level="fail"}` | above 0 |
|
||||
| `llmgw_gpu_nvlink_joined` | 0 |
|
||||
| `llmgw_imex_up` | 0 |
|
||||
|
||||
## What an engine is given
|
||||
|
||||
For a shard on an InfiniBand or RoCE fabric:
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others |
|
||||
| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) |
|
||||
| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address |
|
||||
| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address |
|
||||
| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain |
|
||||
|
||||
A profile's `env` overrides any of them. vLLM launches a multi-node replica
|
||||
with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default):
|
||||
every node runs `vllm serve` with its rank and the leader's address, the
|
||||
workers headless. `launch_mode: ray` is there for a vLLM too old for that.
|
||||
|
||||
## When a replica spanning nodes will not run
|
||||
|
||||
| It | Look at |
|
||||
|---|---|
|
||||
| is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
|
||||
| starts and hangs on its first collective | the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
|
||||
| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
|
||||
| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |
|
||||
+33
-36
@@ -318,32 +318,26 @@ hardware to it.
|
||||
|
||||
### NVLink, if your GPUs have it
|
||||
|
||||
The published binary cannot see NVLink topology. NVML is the only source of it,
|
||||
its bindings need cgo, and the portable binary is built without cgo on purpose
|
||||
so it runs on a node whose libc is older than the builder's.
|
||||
The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
|
||||
joins, each card's NUMA node, and which network adapter is nearest each. A
|
||||
tensor-parallel group is then the set of cards with the most NVLink between
|
||||
them — on a PCIe box where bridges pair specific cards, a group straddling two
|
||||
bridges falls back to PCIe and runs several times slower, and nothing else
|
||||
would report it.
|
||||
|
||||
Without that topology, a tensor-parallel group takes the lowest free GPU
|
||||
indices. On an SXM board with NVSwitch that is correct, because every card is
|
||||
equidistant. On a PCIe box where NVLink bridges pair specific cards, a group
|
||||
straddling two bridges falls back to PCIe and runs several times slower — and
|
||||
nothing reports it, because from the outside it is simply a replica that is
|
||||
slow.
|
||||
|
||||
For those nodes, install the `-nvml` build published beside the portable one:
|
||||
On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
|
||||
driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
|
||||
NVLink Switch system, the GPUs of several nodes form one domain, and a replica
|
||||
can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
|
||||
IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
|
||||
checks it:
|
||||
|
||||
```
|
||||
sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw
|
||||
sudo llmgw doctor
|
||||
```
|
||||
|
||||
It is dynamically linked, so it needs a libc no older than the builder's and the
|
||||
NVIDIA driver present. Check it took effect with:
|
||||
|
||||
```
|
||||
llmgw ctl nodes --json | grep -c nvlink_peers # 0 means the topology is unknown
|
||||
```
|
||||
|
||||
`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a
|
||||
flat zero everywhere means the topology is unknown rather than absent.
|
||||
A `-nvml` build is still published beside the portable one; it reads NVLink
|
||||
from NVML directly, and needs a libc no older than the builder's.
|
||||
|
||||
### InfiniBand, if you have it
|
||||
|
||||
@@ -353,17 +347,17 @@ ibstat # must list your HCAs with State: Active
|
||||
ls /sys/class/infiniband
|
||||
```
|
||||
|
||||
The agent checks what you declare against what the node has. An adapter that is
|
||||
not there, or that has no ACTIVE port — physically linked but with no subnet
|
||||
manager, which survives a reboot and looks like a software problem — takes that
|
||||
fabric out of service on this node, loudly, in the log. Without that check the
|
||||
replica starts, passes its health checks, and hangs on its first all-reduce.
|
||||
Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
|
||||
should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
|
||||
node, and llmgw checks the name against what the adapters report: every port on
|
||||
one fabric sees the same subnet managers, so two independent fabrics given one
|
||||
name are placed as two, and the cluster page says so. An adapter that is not
|
||||
there, or whose port goes down, takes that fabric off the node within seconds,
|
||||
and gives it back when the port returns.
|
||||
|
||||
Fabric membership is configuration, not discovery, and there is no way around
|
||||
that: OpenSM's default subnet prefix is the same on every fabric, so a matching
|
||||
prefix proves nothing and a wrong guess produces a replica that starts, passes
|
||||
health checks, and then hangs on its first all-reduce. You name the fabric when
|
||||
you onboard the node, and two nodes are only in the same fabric if you said so.
|
||||
[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
|
||||
fabric pages show, and what to look at when a replica spanning nodes will not
|
||||
run. Run `llmgw doctor` on the node before it joins.
|
||||
|
||||
### An engine
|
||||
|
||||
@@ -1048,6 +1042,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
and what belongs on which port
|
||||
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
|
||||
each one assumes about this system
|
||||
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
|
||||
checking a node, and what to look at when a replica spanning nodes will not run
|
||||
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
|
||||
deliberately does not do
|
||||
- `docs/onboarding.md` — giving people accounts and keys
|
||||
@@ -1069,8 +1065,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
well-trodden path.
|
||||
- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
|
||||
the fabric invariant and the rendezvous are exercised against simulated
|
||||
hardware only. Single-node placement on real GPUs is the well-trodden path;
|
||||
if you are the first to run a TP group across real cards, expect to find
|
||||
something.
|
||||
- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
|
||||
scheduling, the fabric identity, the maps and the launch are written against
|
||||
the real tools' formats and tested against them, not yet run on hardware that
|
||||
has them. Single-node placement on real GPUs is the well-trodden path; if you
|
||||
are the first to run a TP group across real cards, run `llmgw doctor --json`
|
||||
on each node first and keep the reports.
|
||||
+12
-6
@@ -178,18 +178,24 @@ was cut short.
|
||||
|
||||
## Fabrics
|
||||
|
||||
`--fabric` is how a node says which InfiniBand fabric it is attached to, and
|
||||
there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
|
||||
every fabric, so a matching prefix proves nothing and a wrong guess produces a
|
||||
replica that starts, passes health checks, and then hangs on its first
|
||||
all-reduce.
|
||||
`--fabric` is how a node says which InfiniBand fabric it is attached to. The
|
||||
name is checked against what the adapters report — every port on one fabric
|
||||
sees the same subnet managers — so two independent fabrics given one name are
|
||||
placed as two, and the cluster page says so; give each fabric its own name.
|
||||
The agent installs `infiniband-diags`, which that check needs, if it is missing.
|
||||
|
||||
Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the
|
||||
subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines,
|
||||
and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the
|
||||
rest.
|
||||
|
||||
Omit it for a node with no fabric. Such a node is its own island: it can host
|
||||
single-node replicas and will never be given a multi-node one.
|
||||
|
||||
Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
|
||||
explicitly. An adapter must never appear under two fabrics — the agent refuses
|
||||
that configuration, because the consequence is the hang above.
|
||||
that configuration: a replica given adapters on two fabrics starts, passes its
|
||||
health checks, and hangs on its first all-reduce.
|
||||
|
||||
## Where the driver is
|
||||
|
||||
|
||||
+26
-3
@@ -47,9 +47,32 @@ llmgw users approve ayse@example.com --project Acme/ml/training --store … #
|
||||
```
|
||||
|
||||
That is the whole operator loop. From then on they sign in at `/login`, and
|
||||
`/profile` is where they create keys, see when each was last used, and revoke the
|
||||
ones they are done with. A key is shown once on the page that creates it and
|
||||
never again.
|
||||
their account page (`/profile`) is where they create keys, see when each was
|
||||
last used, revoke the ones they are done with, and see their own use over the
|
||||
last thirty days against every budget that applies to them. A key is shown once
|
||||
on the page that creates it and never again.
|
||||
|
||||
### Connecting an editor
|
||||
|
||||
The account page's **Connect your editor** makes a key for Continue, Codex or
|
||||
Cline and shows what the tool needs with that key, this gateway's address and a
|
||||
model already in it:
|
||||
|
||||
- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
|
||||
so their `config.yaml` is not touched.
|
||||
- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
|
||||
`codex --profile llmgw`; their default model stays as it was. Codex speaks the
|
||||
Responses API, which the gateway translates at `/v1/responses`.
|
||||
- **Cline** — four settings to paste into its own screen, since it keeps them
|
||||
in VS Code's secret storage where no file reaches.
|
||||
|
||||
For Continue and Codex there is a one-line command,
|
||||
`curl -fsSL <gateway>/setup/<tool>?model=… | sh`. Its script is served without
|
||||
signing in and holds no secret — the key comes from the environment — and it
|
||||
keeps a copy of anything it replaces. Revoking the key disconnects the tool.
|
||||
|
||||
How well an editor's agent mode works depends on the model: it needs one that
|
||||
calls tools well, which the coder models do and small general ones often do not.
|
||||
|
||||
**This gateway sends no email.** There is no "your account was approved" message
|
||||
and no password reset, so tell people what to expect: they sign up, they tell
|
||||
|
||||
+3
-2
@@ -793,8 +793,9 @@ IBRoot is the sysfs InfiniBand directory.
|
||||
|
||||
`bool`
|
||||
|
||||
PreferNVML uses the NVML backend when this binary has one. It is the only
|
||||
source of NVLink topology.
|
||||
PreferNVML uses the NVML backend when this binary has one. NVLink
|
||||
topology is read from nvidia-smi topo -m on every build; NVML, where it
|
||||
is used, is the more exact source of the peers.
|
||||
|
||||
### `skip_gpus`
|
||||
|
||||
|
||||
@@ -21,8 +21,9 @@ type Options struct {
|
||||
NvidiaSMI string `koanf:"nvidia_smi"`
|
||||
// IBRoot is the sysfs InfiniBand directory.
|
||||
IBRoot string `koanf:"ib_root"`
|
||||
// PreferNVML uses the NVML backend when this binary has one. It is the only
|
||||
// source of NVLink topology.
|
||||
// PreferNVML uses the NVML backend when this binary has one. NVLink
|
||||
// topology is read from nvidia-smi topo -m on every build; NVML, where it
|
||||
// is used, is the more exact source of the peers.
|
||||
PreferNVML bool `koanf:"prefer_nvml"`
|
||||
// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
|
||||
// found to have none, so this is for one whose driver is broken.
|
||||
|
||||
@@ -36,9 +36,8 @@ func TestTheDocumentedAgentExampleLoads(t *testing.T) {
|
||||
if len(f.HCAs) == 0 {
|
||||
t.Error("the fabric's adapters did not bind")
|
||||
}
|
||||
if f.RateGbps == 0 {
|
||||
t.Error("the fabric's rate did not bind")
|
||||
}
|
||||
// The rate is not documented: the agent measures it, as the slowest of
|
||||
// the fabric's active ports, and a figure written here would be ignored.
|
||||
// node_id is left to the hostname, so the example need not say it; where
|
||||
// the control plane is, nothing can find out.
|
||||
if cfg.Server == "" {
|
||||
|
||||
Reference in new issue
Block a user