Ilker ManapandClaude Opus 5
c13c567cc5
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
96 lines
3.5 KiB
YAML
96 lines
3.5 KiB
YAML
# llmgw agent configuration, one per GPU node.
|
|
#
|
|
# The agent dials out to the control plane and keeps one stream open. Engines
|
|
# keep serving if the agent restarts: the router talks to them directly, and
|
|
# orphan adoption picks them up again on the next start.
|
|
|
|
node_id: gpu-a-01 # defaults to the hostname
|
|
server: "http://llmgw-server.internal:7443"
|
|
|
|
# The agent's half of mutual TLS. The certificate's common name must be this
|
|
# node's id, because the control plane compares the two and refuses the stream
|
|
# when they disagree:
|
|
#
|
|
# llmgw ctl cert issue-client gpu-a-01 --dir /etc/llmgw/pki
|
|
#
|
|
# With this set, `server` above must be an https:// address — an agent holding
|
|
# certificates and dialling http:// is refused at startup rather than connecting
|
|
# in the clear without noticing.
|
|
#
|
|
# tls:
|
|
# ca: /etc/llmgw/ca.pem
|
|
# cert: /etc/llmgw/agent.pem
|
|
# key: /etc/llmgw/agent-key.pem
|
|
|
|
# The address engines are reachable on, and the address NCCL bootstraps over.
|
|
# Never a fabric address: NCCL rendezvouses over TCP before moving to RDMA.
|
|
mgmt_addr: "10.0.0.11"
|
|
mgmt_interface: eth0
|
|
|
|
labels:
|
|
rack: r12
|
|
|
|
heartbeat_interval: 5s
|
|
reconnect_base: 1s
|
|
reconnect_max: 30s
|
|
|
|
# Fabric labelling is explicit and there is no way around it. OpenSM's default
|
|
# subnet prefix is fe80:: on every fabric, so a matching prefix proves nothing
|
|
# and a wrong guess produces a replica that starts, passes health checks and
|
|
# then hangs on its first all-reduce (plan §5.2).
|
|
#
|
|
# A node with no fabrics is its own single-node island: it can host single-node
|
|
# replicas and will never be given a multi-node one.
|
|
fabrics:
|
|
- name: ib-a
|
|
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
|
|
rate_gbps: 200
|
|
|
|
discovery:
|
|
# Worth setting where the driver lives outside the default paths, which
|
|
# containerized driver installs often do.
|
|
nvidia_smi: ""
|
|
ib_root: /sys/class/infiniband
|
|
# NVML is the only source of NVLink topology, which is what keeps a
|
|
# tensor-parallel group off the PCIe path inside a node. It needs a binary
|
|
# built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
|
|
prefer_nvml: true
|
|
skip_gpus: false
|
|
|
|
# Where this node keeps model weights.
|
|
#
|
|
# Unset by default, and unset means the node is not asked to look — which is
|
|
# not the same as having nothing, and constrains nothing. Storage differs at
|
|
# every installation and llmgw deliberately does not own it: a Lustre mount, an
|
|
# image exported over the fabric with SRP or NVMe-oF, local NVMe, or a copy
|
|
# made by hand all look identical from here.
|
|
#
|
|
# What it buys is placement preferring a node that already holds the model over
|
|
# one that must first move tens of gigabytes. That matters most where fabrics
|
|
# are isolated from one another, because weights in one fabric are not weights
|
|
# in another and the only path between them is the management network.
|
|
#
|
|
# model_dirs:
|
|
# - /var/lib/llmgw-models
|
|
# - /mnt/lustre/models
|
|
|
|
engines:
|
|
vllm:
|
|
bin: /opt/venvs/vllm/bin/vllm
|
|
# Which multi-node mechanism works depends on the vLLM version, and the
|
|
# flags have changed more than once. Check the version you deploy.
|
|
launch_mode: ray # "ray" or "mp"
|
|
ray_bin: /opt/venvs/vllm/bin/ray
|
|
sglang:
|
|
# SGLang is launched as a module, which is its documented entry point.
|
|
python: /opt/venvs/sglang/bin/python
|
|
|
|
supervisor:
|
|
work_dir: /var/lib/llmgw-agent
|
|
# Engine output is the only diagnosis when a model fails to load, so keep a
|
|
# useful tail of it.
|
|
log_lines: 512
|
|
# Engines need time to free GPU memory cleanly before being killed.
|
|
stop_timeout: 60s
|
|
health_interval: 2s
|