A fabric's identity was its master subnet manager, which changes on a
failover: for the minute its ports took to agree, one fabric read as
two, and a name covering it as two separate fabrics. Each agent now
also lists every subnet manager on a fabric (saquery SMIR, through one
port per master, once a minute); managers listed together are one
fabric, named by the lowest GUID among them, so the identity survives
the master changing.
The same list says what a fabric can survive. The cluster page's new
subnet manager table, /api/v1/fabrics, doctor and metrics
(llmgw_fabric_master_sm, llmgw_fabric_subnet_managers,
llmgw_fabric_problems) show each fabric's master, priority and
standbys, and say when a fabric has no standby, its ports disagree on
the master, ports are up with no subnet manager (INIT), or the master
changed in the last hour. Every change of master a port reports is
kept, and a fabric's last day of them is listed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A fabric was what agent.yaml called it. With several independent
InfiniBand fabrics at a site, two given one name would have had a
replica spread across both, starting and then hanging on its first
collective. Each active InfiniBand port now reports the master subnet
manager it sees (sminfo, asked once a minute), which every port on one
fabric shares and ports on separate fabrics cannot; a RoCE port reports
the IP network of its RoCE v2 address. Ports also carry their node and
port GUIDs.
Placement divides a declared fabric by that physical identity and forms
a replica on one part only; a node whose adapters for one name are on
two fabrics is refused, and a node that cannot say is kept apart from
those that can. /api/v1/fabrics and the cluster page list each physical
fabric with its subnet manager, and say when one name covers two
fabrics, two names one fabric, an adapter is up but undeclared, or a
node cannot be checked. doctor reports each port's subnet manager, and
an agent on a node with adapters installs infiniband-diags when sminfo
is missing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vLLM's mp launch gave workers no rank and the leader's address as their
own VLLM_HOST_IP; every node now runs serve with --nnodes, --node-rank,
--master-addr and --master-port, and the workers --headless, as vLLM
documents it.
NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME were the control plane's one
interface name for every node; the agent already worked out its own -
the one holding its management address - and now uses it.
On RoCE, NCCL's default GID index is a v1 link-local entry on most hosts,
which does not route. The agent finds the RoCE v2 entry holding an IPv4
address and passes it as NCCL_IB_GID_INDEX; InfiniBand gets none.
A worker whose only process needs the leader - a Ray join - started
without waiting for it, since the wait was honoured for sidecars alone;
it found no head, exited and failed the replica. It now waits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A port's state was read once, at registration: one that went down later
left its node on the fabric, offered multi-node work that would hang on
its first collective, until somebody restarted the agent. The agent now
re-reads every IB and RoCE port on each heartbeat - state, physical
state, rate and the error and data counters - and when the fabrics it
can carry traffic on change, says so once; the control plane replaces
the node's fabrics, which is what placement reads.
The ports are exported as llmgw_ib_port_active (with state and
phys_state, so an unplugged cable reads differently from a missing
subnet manager), llmgw_ib_port_rate_gbps and llmgw_ib_port_counter_total,
and a node's page lists them with the fabric each carries and any error
counter that is not zero.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Everything here works without GPUs and is covered by tests. The control plane
connection is not built yet, so llmgw agent itself is still a stub and the phase
exit criterion is not met.
- discovery: IB from /sys/class/infiniband, parsing the formats sysfs actually
uses and telling native InfiniBand from RoCE; GPUs via nvidia-smi, the
cgo-free path, with health flags for uncorrected ECC and memory already held
by another process. A node without IB is not an error, it is a single-node
island.
- engines: vLLM and SGLang command and environment construction, kept free of
process handling so it stays testable. The NCCL environment is the part the
plan marks critical: NCCL_IB_HCA lists only the chosen fabric's adapters,
because leaving it open lets NCCL pick an adapter on a fabric where the peer
is unreachable, and the job then hangs instead of failing.
- validation before launch: no mixed GPU types in a tensor-parallel group, the
node/GPU/TP arithmetic has to add up, and fp8 is refused below compute
capability 8.9 rather than silently falling back to a slower path.
- supervisor: process groups, because engines fork workers that hold GPU memory
and outlive a signal sent only to the leader; a bounded log tail, since engine
output is the only diagnosis when loading fails; Starting separate from
Loading, because a 70B model legitimately takes minutes; and orphan adoption
through pid files fingerprinted with the command line, since pids get reused
and adopting an unrelated process is worse than leaking the original.
- readiness is two questions: /health says the process is up, /v1/models says
the weights loaded. Routing on /health alone sends traffic into a loading model.
- llmgw agent inventory reports what the scheduler would see on a node.
One plan correction: GPU class sizes come from the driver in GiB, which does not
always match the marketing name. An L40S is sold as 48GB and reports 44.99 GiB,
so its class is l40s-45g, not l40s-48g. Consistency matters more than matching
the box, because that number decides whether a model fits. Plan §2 and §5.1
updated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>