Compare commits
18
Commits
v0.10.6
...
v0.10.7-rc.7
No files matched your search
+256
@@ -20,6 +20,262 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
From the field report in issue #2, after a deployment on GPU nodes.
|
||||
|
||||
### GPU metrics that move, and charts of them
|
||||
|
||||
A GPU's utilisation and memory in use were read once, when the agent
|
||||
started, and published from then on: a card busy since looked as idle as it
|
||||
was at registration, and one idle since as busy. They are now sampled with
|
||||
the temperatures and power every heartbeat (ten seconds at most), with the
|
||||
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
|
||||
An agent too old to send them still has its registration figures published,
|
||||
and a figure the driver does not give is left out rather than shown as zero.
|
||||
|
||||
- **New series**: `llmgw_gpu_memory_utilization_percent`,
|
||||
`llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
|
||||
`llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
|
||||
sample.
|
||||
- **The node page** shows each card's utilisation, memory, clock and ECC
|
||||
errors in its table, and charts of utilisation, memory, temperature and
|
||||
power, one line per card.
|
||||
- **The cluster page** charts the fleet's GPUs by class: average utilisation
|
||||
and memory in use.
|
||||
- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
|
||||
utilisation, memory, memory bandwidth, temperature, power against the
|
||||
limit, clock, and what holds cards back.
|
||||
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
|
||||
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
|
||||
- **The cluster page's GPU map**: every card a square, a row per node,
|
||||
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
|
||||
with what holds it and its figures under the pointer; free cards dashed.
|
||||
- **Cards held and doing nothing**: the replicas whose every card reads under
|
||||
5% busy, with when they were last asked — the first place to find room.
|
||||
- **What each model gets from its cards**, on the cluster page and in
|
||||
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
|
||||
draw and average utilisation, over the last quarter hour.
|
||||
- **What wears a card out**, read every minute: XID errors from the kernel's
|
||||
log (counted from when the agent started), memory rows remapped or pages
|
||||
retired and whether either waits for a reset, a card out of spare rows,
|
||||
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
|
||||
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
|
||||
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
|
||||
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
|
||||
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
|
||||
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
|
||||
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
|
||||
`GPUNVLinkErrorsClimbing`.
|
||||
- **GPU time by owner** on the usage page: the cards each owner's models
|
||||
held over the window, charted and summed into GPU-hours, whether or not the
|
||||
models were asked anything — the time a GPU quota is about. From the
|
||||
metrics store, as far back as it keeps history.
|
||||
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
|
||||
each card and whose quota it counts against, for joining the GPU series to
|
||||
models and tenants.
|
||||
|
||||
### A GPU placement no longer runs on the CPU unnoticed
|
||||
|
||||
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
|
||||
where the loader looks), came up on the CPU, answered at CPU speed, and was
|
||||
reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||||
|
||||
- **Installing** the CUDA build asks it which devices it sees
|
||||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||||
own `ldd` could not say it: the backend is opened at run time.
|
||||
- **After the upgrade**, engines the agent finds already running are held to
|
||||
the same check when it adopts them: one serving a GPU placement on the CPU
|
||||
is stopped, and its replacement starts through the check. The installed
|
||||
CUDA build is checked again within ten minutes, and installed afresh if it
|
||||
sees no GPU.
|
||||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||||
included). One that does not is stopped and reported failed: "placed on GPU
|
||||
0 but computing on the CPU", with the devices the engine lists. This covers
|
||||
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
|
||||
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
|
||||
when the engine sees no GPU, and otherwise prints the devices it computes on.
|
||||
|
||||
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
|
||||
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
|
||||
node with two kinds of GPU, a shard could get a different card from the one it
|
||||
was given.
|
||||
|
||||
### A model of many files, mirrored whole for vLLM
|
||||
|
||||
vLLM loads a directory — shards, config.json, the tokenizer — and does not
|
||||
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
|
||||
shards alone, a node could be handed a version still downloading or missing a
|
||||
shard, and a repository's page given as a model's source was fetched as one
|
||||
file of HTML.
|
||||
|
||||
- **The explorer fetches a safetensors set with what the engine reads beside
|
||||
it**: config.json, generation_config.json, the tokenizer files, the shard
|
||||
index, the quantisation config. Never code: a `.py` would be the engine
|
||||
running what the repository ships, which is to be chosen on purpose.
|
||||
- **A version is offered to nodes only whole.** The mirror refuses its
|
||||
manifest (409, with what is missing) while a file is still downloading, or
|
||||
while a shard its `model.safetensors.index.json` names, or a part of a
|
||||
split (`-00002-of-00004`), is absent. A node reports "not complete on the
|
||||
mirror yet" and is tried again.
|
||||
- **Checksums go with the files.** The mirror keeps each file's sha256 beside
|
||||
it and the manifest carries it; a node checks every file of a set against
|
||||
it before using the set, and fetches one that differs again.
|
||||
- **`llmgw mirror add` takes several URLs** into one version.
|
||||
- **A Hugging Face repository page given as a source** is read as the
|
||||
repository id for a model vLLM or SGLang runs, which the engine downloads
|
||||
itself (the node then needs to reach Hugging Face), and refused for one
|
||||
llama.cpp runs, which needs a file.
|
||||
|
||||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||||
such a model, as before.
|
||||
|
||||
### Signing up with an address that has an account
|
||||
|
||||
On PostgreSQL, signing up with an address already registered showed
|
||||
"something went wrong; try again": the store recognised a duplicate by the
|
||||
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
|
||||
code 23505. It now knows both, and the address is answered as on SQLite —
|
||||
with the same page as a new sign-up, so the page cannot be used to find out
|
||||
who has an account. That page now also says what to do if the address had
|
||||
one: sign in, or ask an operator to reset the password.
|
||||
|
||||
An error the page does not explain says the account was not created and
|
||||
gives a reference (`err-…`), logged beside the error, for an operator to find
|
||||
it; the database's own words stay in the log.
|
||||
|
||||
### An adapter no fabric uses does not keep a node out
|
||||
|
||||
`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
|
||||
`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
|
||||
added for single-node inference, with no fabric given, that refused it for
|
||||
RDMA nobody had asked for.
|
||||
|
||||
- Those are failures only for ports a declared fabric uses, and only when a
|
||||
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
|
||||
- Which fabric an undeclared InfiniBand port is on, and whether it has a
|
||||
subnet manager LID, are notes rather than warnings; `node add` still warns
|
||||
once that adapters are up with no fabric given.
|
||||
- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
|
||||
name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
|
||||
written afterwards.
|
||||
|
||||
### A restart with no room beside it
|
||||
|
||||
`ctl restart` places each replacement beside the replica it replaces. With
|
||||
no room for it — one GPU, every card that fits held by the replica itself —
|
||||
the old replica was stopped only after the drain grace period, five minutes
|
||||
of "restart replacing" with nothing saying why, which read as a restart that
|
||||
never happened; scaling to zero and back was the way out. And a replacement
|
||||
that fit nowhere even with the old replica gone was stopped for all the same,
|
||||
leaving the model with nothing. Now:
|
||||
|
||||
- **The replica is the only thing in the way**, found by placing the
|
||||
replacement on a copy of the inventory without it: stopped at once, the
|
||||
replacement taking its place.
|
||||
- **The replacement fits nowhere even then**: the restart is given up and the
|
||||
replica keeps serving. `ctl replicas` shows `restart refused:` and why.
|
||||
- **Room being made** — a lower class preempted, an idle on-demand replica
|
||||
stopping — is waited for, as before.
|
||||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||||
|
||||
### Every GPU node gets engine builds it can run, by itself
|
||||
|
||||
Which build of an engine a node installs follows its hardware — the CPU
|
||||
architecture, the driver, and the oldest GPU on it — from what each project
|
||||
publishes and how it was compiled. Each engine has a list of builds, best
|
||||
first; the node takes the first its hardware meets, and the first of those
|
||||
that installs and is seen to compute on the GPU. A build the table is wrong
|
||||
about costs its download, and the next one is tried.
|
||||
|
||||
| | llama.cpp | vLLM |
|
||||
|---|---|---|
|
||||
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
|
||||
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
|
||||
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
|
||||
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
|
||||
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
|
||||
| No NVIDIA GPU | the CPU build | none |
|
||||
|
||||
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
|
||||
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
|
||||
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
|
||||
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
|
||||
starts at 7.5.
|
||||
|
||||
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
|
||||
column shows the version installed, and the installation's detail says why
|
||||
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
|
||||
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
|
||||
vLLM is not placed on a V100 by accident.
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation for vLLM, the devices llama.cpp lists for its GPU builds.
|
||||
- **The build can still be chosen per node** on the cluster page (auto, CUDA
|
||||
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
|
||||
at once.
|
||||
- **Whether an engine computes on its GPUs is read from every process
|
||||
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
|
||||
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
|
||||
`CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
### Engines per node, and nodes that do not run them all
|
||||
|
||||
A cluster with older GPUs could not add them cleanly: `node add` refused a
|
||||
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
|
||||
and `--ignore-checks` passed over every other problem with it.
|
||||
|
||||
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
|
||||
node joins without it and serves what it does run. `node add` checks the
|
||||
vLLM the control plane installs, not the binary's own default.
|
||||
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none — for exceptions; the
|
||||
build each GPU generation needs is chosen without it (above).
|
||||
- **The requirements follow the version**, as the table above sets out; the
|
||||
threshold had been CUDA 13's for every version.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
|
||||
### Serving from the mirror says where a model will run
|
||||
|
||||
The serve form put every GGUF model on the CPU: it offered no choice, and its
|
||||
"Profile name" field, pre-filled "cpu", read as though renaming it would move
|
||||
the model to a GPU. Placement is decided by `gpu_class`, which the form never
|
||||
set. Now:
|
||||
|
||||
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
|
||||
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
|
||||
cards is chosen.
|
||||
- **GPUs per replica** sets how many cards a replica takes on one node
|
||||
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
|
||||
how many a llama.cpp replica serves at once.
|
||||
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
|
||||
typed.
|
||||
- A line under the form says what the model will run as — engine, CPU or how
|
||||
many of which GPU, memory, context pool and how it is divided — and turns
|
||||
red when the CPU is chosen with GPUs free. The message after adding repeats
|
||||
it.
|
||||
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
|
||||
anyway** is ticked.
|
||||
|
||||
### llama.cpp on several GPUs of one node
|
||||
|
||||
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
|
||||
placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||||
placed — the replica stayed unplaced or ran on one card. It is now read as
|
||||
`gpus_per_node`, so a profile written that way is placed on that many cards
|
||||
after the upgrade, rather than dropped; only both given and disagreeing, or
|
||||
`nodes` above one, is refused.
|
||||
|
||||
## v0.10.6 — 2026-10-01
|
||||
|
||||
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
|
||||
|
||||
@@ -1550,7 +1550,36 @@ type GPUSensor struct {
|
||||
PowerLimitW float64 `protobuf:"fixed64,5,opt,name=power_limit_w,json=powerLimitW,proto3" json:"power_limit_w,omitempty"`
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
|
||||
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
|
||||
// How busy the card is, sampled with the rest, rather than read once at
|
||||
// registration. -1 is not reported by the driver. memory_total_mb is set
|
||||
// by every agent that sends these, so zero there is an older agent.
|
||||
UtilizationPercent float64 `protobuf:"fixed64,7,opt,name=utilization_percent,json=utilizationPercent,proto3" json:"utilization_percent,omitempty"`
|
||||
MemoryUtilizationPercent float64 `protobuf:"fixed64,8,opt,name=memory_utilization_percent,json=memoryUtilizationPercent,proto3" json:"memory_utilization_percent,omitempty"`
|
||||
MemoryUsedMb float64 `protobuf:"fixed64,9,opt,name=memory_used_mb,json=memoryUsedMb,proto3" json:"memory_used_mb,omitempty"`
|
||||
MemoryTotalMb float64 `protobuf:"fixed64,10,opt,name=memory_total_mb,json=memoryTotalMb,proto3" json:"memory_total_mb,omitempty"`
|
||||
SmClockMhz float64 `protobuf:"fixed64,11,opt,name=sm_clock_mhz,json=smClockMhz,proto3" json:"sm_clock_mhz,omitempty"`
|
||||
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
|
||||
EccUncorrected float64 `protobuf:"fixed64,12,opt,name=ecc_uncorrected,json=eccUncorrected,proto3" json:"ecc_uncorrected,omitempty"`
|
||||
// What wears the card out, read every minute; has_wear says these are
|
||||
// present. Counts of -1 are not reported for this card.
|
||||
HasWear bool `protobuf:"varint,13,opt,name=has_wear,json=hasWear,proto3" json:"has_wear,omitempty"`
|
||||
PcieReplays int64 `protobuf:"varint,14,opt,name=pcie_replays,json=pcieReplays,proto3" json:"pcie_replays,omitempty"`
|
||||
RetiredPagesSbe int64 `protobuf:"varint,15,opt,name=retired_pages_sbe,json=retiredPagesSbe,proto3" json:"retired_pages_sbe,omitempty"`
|
||||
RetiredPagesDbe int64 `protobuf:"varint,16,opt,name=retired_pages_dbe,json=retiredPagesDbe,proto3" json:"retired_pages_dbe,omitempty"`
|
||||
RetirementPending bool `protobuf:"varint,17,opt,name=retirement_pending,json=retirementPending,proto3" json:"retirement_pending,omitempty"`
|
||||
RemappedRowsCorrectable int64 `protobuf:"varint,18,opt,name=remapped_rows_correctable,json=remappedRowsCorrectable,proto3" json:"remapped_rows_correctable,omitempty"`
|
||||
RemappedRowsUncorrectable int64 `protobuf:"varint,19,opt,name=remapped_rows_uncorrectable,json=remappedRowsUncorrectable,proto3" json:"remapped_rows_uncorrectable,omitempty"`
|
||||
RemapPending bool `protobuf:"varint,20,opt,name=remap_pending,json=remapPending,proto3" json:"remap_pending,omitempty"`
|
||||
RemapFailure bool `protobuf:"varint,21,opt,name=remap_failure,json=remapFailure,proto3" json:"remap_failure,omitempty"`
|
||||
NvlinkCrcErrors int64 `protobuf:"varint,22,opt,name=nvlink_crc_errors,json=nvlinkCrcErrors,proto3" json:"nvlink_crc_errors,omitempty"`
|
||||
NvlinkReplayErrors int64 `protobuf:"varint,23,opt,name=nvlink_replay_errors,json=nvlinkReplayErrors,proto3" json:"nvlink_replay_errors,omitempty"`
|
||||
NvlinkRecoveryErrors int64 `protobuf:"varint,24,opt,name=nvlink_recovery_errors,json=nvlinkRecoveryErrors,proto3" json:"nvlink_recovery_errors,omitempty"`
|
||||
// xid_errors are the driver's error events since the agent started.
|
||||
XidErrors int64 `protobuf:"varint,25,opt,name=xid_errors,json=xidErrors,proto3" json:"xid_errors,omitempty"`
|
||||
LastXid int32 `protobuf:"varint,26,opt,name=last_xid,json=lastXid,proto3" json:"last_xid,omitempty"`
|
||||
LastXidText string `protobuf:"bytes,27,opt,name=last_xid_text,json=lastXidText,proto3" json:"last_xid_text,omitempty"`
|
||||
LastXidUnix int64 `protobuf:"varint,28,opt,name=last_xid_unix,json=lastXidUnix,proto3" json:"last_xid_unix,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -1627,6 +1656,160 @@ func (x *GPUSensor) GetThrottle() []string {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetUtilizationPercent() float64 {
|
||||
if x != nil {
|
||||
return x.UtilizationPercent
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryUtilizationPercent() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryUtilizationPercent
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryUsedMb() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryUsedMb
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryTotalMb() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryTotalMb
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetSmClockMhz() float64 {
|
||||
if x != nil {
|
||||
return x.SmClockMhz
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetEccUncorrected() float64 {
|
||||
if x != nil {
|
||||
return x.EccUncorrected
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetHasWear() bool {
|
||||
if x != nil {
|
||||
return x.HasWear
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetPcieReplays() int64 {
|
||||
if x != nil {
|
||||
return x.PcieReplays
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetiredPagesSbe() int64 {
|
||||
if x != nil {
|
||||
return x.RetiredPagesSbe
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetiredPagesDbe() int64 {
|
||||
if x != nil {
|
||||
return x.RetiredPagesDbe
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetirementPending() bool {
|
||||
if x != nil {
|
||||
return x.RetirementPending
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemappedRowsCorrectable() int64 {
|
||||
if x != nil {
|
||||
return x.RemappedRowsCorrectable
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemappedRowsUncorrectable() int64 {
|
||||
if x != nil {
|
||||
return x.RemappedRowsUncorrectable
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemapPending() bool {
|
||||
if x != nil {
|
||||
return x.RemapPending
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemapFailure() bool {
|
||||
if x != nil {
|
||||
return x.RemapFailure
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkCrcErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkCrcErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkReplayErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkReplayErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkRecoveryErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkRecoveryErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetXidErrors() int64 {
|
||||
if x != nil {
|
||||
return x.XidErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXid() int32 {
|
||||
if x != nil {
|
||||
return x.LastXid
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXidText() string {
|
||||
if x != nil {
|
||||
return x.LastXidText
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXidUnix() int64 {
|
||||
if x != nil {
|
||||
return x.LastXidUnix
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
type ShardLoad struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -2370,9 +2553,12 @@ func (x *Provision) GetEngines() []*EngineWant {
|
||||
|
||||
// EngineWant names one engine at one version.
|
||||
type EngineWant struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
|
||||
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
|
||||
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
|
||||
// variant is the build to install, when one is chosen for the node rather
|
||||
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
|
||||
Variant string `protobuf:"bytes,3,opt,name=variant,proto3" json:"variant,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -2421,6 +2607,13 @@ func (x *EngineWant) GetVersion() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *EngineWant) GetVariant() string {
|
||||
if x != nil {
|
||||
return x.Variant
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ProvisionStatus is where one engine's installation stands on a node.
|
||||
type ProvisionStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -3989,14 +4182,39 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
" \x01(\x01R\tcpuPowerW\x12 \n" +
|
||||
"\finlet_temp_c\x18\v \x01(\x01R\n" +
|
||||
"inletTempC\x12'\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xb5\x01\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xde\b\n" +
|
||||
"\tGPUSensor\x12\x14\n" +
|
||||
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x15\n" +
|
||||
"\x06temp_c\x18\x02 \x01(\x01R\x05tempC\x12\"\n" +
|
||||
"\rmemory_temp_c\x18\x03 \x01(\x01R\vmemoryTempC\x12\x17\n" +
|
||||
"\apower_w\x18\x04 \x01(\x01R\x06powerW\x12\"\n" +
|
||||
"\rpower_limit_w\x18\x05 \x01(\x01R\vpowerLimitW\x12\x1a\n" +
|
||||
"\bthrottle\x18\x06 \x03(\tR\bthrottle\"m\n" +
|
||||
"\bthrottle\x18\x06 \x03(\tR\bthrottle\x12/\n" +
|
||||
"\x13utilization_percent\x18\a \x01(\x01R\x12utilizationPercent\x12<\n" +
|
||||
"\x1amemory_utilization_percent\x18\b \x01(\x01R\x18memoryUtilizationPercent\x12$\n" +
|
||||
"\x0ememory_used_mb\x18\t \x01(\x01R\fmemoryUsedMb\x12&\n" +
|
||||
"\x0fmemory_total_mb\x18\n" +
|
||||
" \x01(\x01R\rmemoryTotalMb\x12 \n" +
|
||||
"\fsm_clock_mhz\x18\v \x01(\x01R\n" +
|
||||
"smClockMhz\x12'\n" +
|
||||
"\x0fecc_uncorrected\x18\f \x01(\x01R\x0eeccUncorrected\x12\x19\n" +
|
||||
"\bhas_wear\x18\r \x01(\bR\ahasWear\x12!\n" +
|
||||
"\fpcie_replays\x18\x0e \x01(\x03R\vpcieReplays\x12*\n" +
|
||||
"\x11retired_pages_sbe\x18\x0f \x01(\x03R\x0fretiredPagesSbe\x12*\n" +
|
||||
"\x11retired_pages_dbe\x18\x10 \x01(\x03R\x0fretiredPagesDbe\x12-\n" +
|
||||
"\x12retirement_pending\x18\x11 \x01(\bR\x11retirementPending\x12:\n" +
|
||||
"\x19remapped_rows_correctable\x18\x12 \x01(\x03R\x17remappedRowsCorrectable\x12>\n" +
|
||||
"\x1bremapped_rows_uncorrectable\x18\x13 \x01(\x03R\x19remappedRowsUncorrectable\x12#\n" +
|
||||
"\rremap_pending\x18\x14 \x01(\bR\fremapPending\x12#\n" +
|
||||
"\rremap_failure\x18\x15 \x01(\bR\fremapFailure\x12*\n" +
|
||||
"\x11nvlink_crc_errors\x18\x16 \x01(\x03R\x0fnvlinkCrcErrors\x120\n" +
|
||||
"\x14nvlink_replay_errors\x18\x17 \x01(\x03R\x12nvlinkReplayErrors\x124\n" +
|
||||
"\x16nvlink_recovery_errors\x18\x18 \x01(\x03R\x14nvlinkRecoveryErrors\x12\x1d\n" +
|
||||
"\n" +
|
||||
"xid_errors\x18\x19 \x01(\x03R\txidErrors\x12\x19\n" +
|
||||
"\blast_xid\x18\x1a \x01(\x05R\alastXid\x12\"\n" +
|
||||
"\rlast_xid_text\x18\x1b \x01(\tR\vlastXidText\x12\"\n" +
|
||||
"\rlast_xid_unix\x18\x1c \x01(\x03R\vlastXidUnix\"m\n" +
|
||||
"\tShardLoad\x12\x19\n" +
|
||||
"\bshard_id\x18\x01 \x01(\tR\ashardId\x12\x1b\n" +
|
||||
"\tcpu_cores\x18\x02 \x01(\x01R\bcpuCores\x12(\n" +
|
||||
@@ -4068,11 +4286,12 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
||||
"\tProvision\x12.\n" +
|
||||
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\">\n" +
|
||||
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\"X\n" +
|
||||
"\n" +
|
||||
"EngineWant\x12\x16\n" +
|
||||
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\"\x8b\x01\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
|
||||
"\avariant\x18\x03 \x01(\tR\avariant\"\x8b\x01\n" +
|
||||
"\x0fProvisionStatus\x12\x16\n" +
|
||||
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
|
||||
|
||||
@@ -304,6 +304,35 @@ message GPUSensor {
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
repeated string throttle = 6;
|
||||
// How busy the card is, sampled with the rest, rather than read once at
|
||||
// registration. -1 is not reported by the driver. memory_total_mb is set
|
||||
// by every agent that sends these, so zero there is an older agent.
|
||||
double utilization_percent = 7;
|
||||
double memory_utilization_percent = 8;
|
||||
double memory_used_mb = 9;
|
||||
double memory_total_mb = 10;
|
||||
double sm_clock_mhz = 11;
|
||||
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
|
||||
double ecc_uncorrected = 12;
|
||||
// What wears the card out, read every minute; has_wear says these are
|
||||
// present. Counts of -1 are not reported for this card.
|
||||
bool has_wear = 13;
|
||||
int64 pcie_replays = 14;
|
||||
int64 retired_pages_sbe = 15;
|
||||
int64 retired_pages_dbe = 16;
|
||||
bool retirement_pending = 17;
|
||||
int64 remapped_rows_correctable = 18;
|
||||
int64 remapped_rows_uncorrectable = 19;
|
||||
bool remap_pending = 20;
|
||||
bool remap_failure = 21;
|
||||
int64 nvlink_crc_errors = 22;
|
||||
int64 nvlink_replay_errors = 23;
|
||||
int64 nvlink_recovery_errors = 24;
|
||||
// xid_errors are the driver's error events since the agent started.
|
||||
int64 xid_errors = 25;
|
||||
int32 last_xid = 26;
|
||||
string last_xid_text = 27;
|
||||
int64 last_xid_unix = 28;
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
@@ -465,6 +494,9 @@ message Provision {
|
||||
message EngineWant {
|
||||
string engine = 1; // "llamacpp"
|
||||
string version = 2; // "b10991"
|
||||
// variant is the build to install, when one is chosen for the node rather
|
||||
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
|
||||
string variant = 3;
|
||||
}
|
||||
|
||||
// ProvisionStatus is where one engine's installation stands on a node.
|
||||
|
||||
@@ -487,6 +487,427 @@
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "GPUs",
|
||||
"id": 14,
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 30,
|
||||
"w": 24,
|
||||
"h": 1
|
||||
},
|
||||
"collapsed": false,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU busy, by card",
|
||||
"description": "Share of each sample period a kernel was running, sampled every heartbeat.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 31,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 15,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_utilization_percent",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU memory in use, by card",
|
||||
"description": "Framebuffer in use. Engines reserve most of it at start.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 31,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 16,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_memory_used_bytes / llmgw_gpu_memory_total_bytes",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU memory bandwidth busy, by card",
|
||||
"description": "Decode is bound by memory: high here with compute low is a card waiting on memory.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 38,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 17,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_memory_utilization_percent",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU temperature",
|
||||
"description": "Throttling usually starts in the high 80s.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 38,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 18,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_temperature_celsius",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "celsius"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU power against its limit",
|
||||
"description": "1 is the enforced limit; a card held there is power-capped.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 45,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 19,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1.1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU clock",
|
||||
"description": "A clock held low under load is the card throttled; see the next panel for why.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 45,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 20,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_sm_clock_mhz",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "rotmhz"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "What holds GPUs back",
|
||||
"description": "Cards throttled now, by reason, and uncorrected ECC errors since the driver loaded.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 52,
|
||||
"w": 24,
|
||||
"h": 7
|
||||
},
|
||||
"id": 21,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (reason) (llmgw_gpu_throttled)",
|
||||
"legendFormat": "throttled: {{reason}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum by (node, gpu) (llmgw_gpu_ecc_errors)",
|
||||
"legendFormat": "ECC errors {{node}} gpu {{gpu}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Output tokens/s per GPU, by model",
|
||||
"description": "What each model gets out of each card it holds.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 59,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 22,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) / count by (model) (llmgw_gpu_assigned)",
|
||||
"legendFormat": "{{model}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Output tokens per watt-hour, by model",
|
||||
"description": "Tokens for the energy the model's cards draw.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 59,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 23,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) * 3600 / sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)",
|
||||
"legendFormat": "{{model}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU wear",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"id": 24,
|
||||
"description": "What wears cards out: XID errors in the last 15 minutes, memory rows remapped, PCIe retries an hour. Anything above zero here is worth a look on that card's node page.",
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 66,
|
||||
"w": 24,
|
||||
"h": 7
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_xid_errors[15m]) > 0",
|
||||
"legendFormat": "XID {{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum by (node, gpu) (llmgw_gpu_remapped_rows) > 0",
|
||||
"legendFormat": "rows remapped {{node}} gpu {{gpu}}",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_pcie_replays[1h]) > 0",
|
||||
"legendFormat": "PCIe retries/h {{node}} gpu {{gpu}}",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_nvlink_errors[1h]) > 0",
|
||||
"legendFormat": "NVLink {{kind}}/h {{node}} gpu {{gpu}}",
|
||||
"refId": "D"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -200,6 +200,94 @@ groups:
|
||||
is skipped by placement rather than failing, so this is quiet until
|
||||
somebody wonders why a model will not land there.
|
||||
|
||||
- name: llmgw-gpus
|
||||
rules:
|
||||
- alert: GPUThermalThrottling
|
||||
expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})"
|
||||
description: >-
|
||||
Ten minutes held back by temperature: every replica on the card is
|
||||
slower than its measurements say. Usually airflow, a fan, or the
|
||||
room — the node page shows the card's temperature over time.
|
||||
|
||||
- alert: GPUUncorrectedECCErrors
|
||||
expr: llmgw_gpu_ecc_errors > 0
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors"
|
||||
description: >-
|
||||
Uncorrected memory errors since the driver loaded: what runs on the
|
||||
card may be computing on corrupted memory. Drain the node
|
||||
(llmgw ctl drain) and have the card looked at.
|
||||
|
||||
- alert: GPUXidError
|
||||
expr: increase(llmgw_gpu_xid_errors[15m]) > 0
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
|
||||
description: >-
|
||||
The driver reported an error on the card (its code is
|
||||
llmgw_gpu_last_xid, and the node page says it in words). 79 is a
|
||||
card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
|
||||
94/95 contained and uncontained ECC errors: those need the card
|
||||
looked at. Others are often an engine's own fault, and pass.
|
||||
|
||||
- alert: GPUOutOfSpareRows
|
||||
expr: llmgw_gpu_remap_failure == 1
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
|
||||
description: >-
|
||||
The card cannot remap another failing row. Drain the node and
|
||||
have the card replaced.
|
||||
|
||||
- alert: GPUNeedsReset
|
||||
expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
|
||||
description: >-
|
||||
Rows remapped or pages retired take effect only after the card is
|
||||
reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
|
||||
it back.
|
||||
|
||||
- alert: GPUPCIeRetriesClimbing
|
||||
expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
|
||||
description: >-
|
||||
The card's link to the host is retrying. A riser, the slot or a
|
||||
cable is the usual cause, and transfers to the card slow down.
|
||||
|
||||
- alert: GPUNVLinkErrorsClimbing
|
||||
expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
|
||||
description: >-
|
||||
Errors on the card's NVLink links: tensor-parallel replicas on it
|
||||
slow down, and a link that keeps failing takes a replica with it.
|
||||
|
||||
- alert: GPUHot
|
||||
expr: llmgw_gpu_temperature_celsius > 87
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C"
|
||||
description: >-
|
||||
Ten minutes above 87 °C, where most data-centre cards begin to
|
||||
throttle.
|
||||
|
||||
- name: monitoring-host
|
||||
rules:
|
||||
- alert: MonitoringDiskFilling
|
||||
|
||||
+67
-9
@@ -367,7 +367,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
|
||||
| Engine | On | How |
|
||||
|---|---|---|
|
||||
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
|
||||
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
|
||||
@@ -376,8 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
|
||||
vLLM is built for CUDA 13. A V100 cannot run it;
|
||||
- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
|
||||
the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
|
||||
CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
|
||||
publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
|
||||
(7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
|
||||
**vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
|
||||
kernels — and the cluster page shows it running that version; run `llmgw
|
||||
ctl engine-check vllm` there once, since llmgw's flags are checked against
|
||||
the current one. Pascal and older run no vLLM. Whatever is chosen, the
|
||||
install ends by computing something on the GPU; a build that cannot is
|
||||
passed over for the next;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -394,8 +403,51 @@ engines:
|
||||
```
|
||||
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself instead, set its version to `""` and name its path in
|
||||
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
|
||||
you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
|
||||
build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
|
||||
*CUDA 12.9*, or *none* — or from the command line:
|
||||
|
||||
```
|
||||
llmgw ctl engine-build gpu07 vllm cu129
|
||||
```
|
||||
|
||||
The choice is kept by the control plane and the node is told at once; it
|
||||
installs the build in the background, and the column shows how that stands.
|
||||
|
||||
#### Different GPUs, different engines
|
||||
|
||||
A cluster of two GPU generations may need two builds of an engine, or one on
|
||||
only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
|
||||
id, or by labels, every one given having to match — and lays engines over the
|
||||
cluster's for them, in order:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: a10g # pin a class to a version you have tested
|
||||
engines: { vllm: "0.29.0" }
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
|
||||
A node told nothing it can run of an engine joins without it: `llmgw doctor`
|
||||
and `node add` warn, and the node serves what it does run. To keep an engine
|
||||
off a node for good, whatever the control plane says, add the node with
|
||||
`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
|
||||
into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
|
||||
than `--ignore-checks`, which passes over every other problem as well.
|
||||
|
||||
A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
|
||||
asks it which devices it sees, and a replica placed on GPUs is not ready until
|
||||
its process holds memory on each of them. A build missing a CUDA library fails
|
||||
there, naming the library, rather than serving on the CPU.
|
||||
|
||||
After changing a version, check it still takes the flags llmgw builds for it,
|
||||
which drift between releases:
|
||||
@@ -1032,7 +1084,12 @@ A replica may come back marked **`ready (stale profile)`**. That means it is
|
||||
running a profile that has since been edited — the reconciler leaves a serving
|
||||
replica alone on purpose, because churning one costs more than the difference.
|
||||
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
|
||||
the new one before stopping the old.
|
||||
the new one before stopping the old. Where the old one is all that stands in
|
||||
the new one's way — a single GPU — it is stopped first, at once, and the model
|
||||
is unserved while the new one loads; where the new profile fits nowhere even
|
||||
then, the restart is refused and the old one keeps serving, which `ctl
|
||||
replicas` shows as `restart refused:` with the reason. `--in-place` asks for
|
||||
stop-then-start whatever room there is.
|
||||
|
||||
### Signed releases
|
||||
|
||||
@@ -1216,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
|
||||
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
|
||||
install a vLLM built for your CUDA yourself.
|
||||
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
|
||||
no build from 0.20 on has Volta kernels, and the node picks the newest that
|
||||
has. llmgw's flags are checked against the current version, so run `llmgw
|
||||
ctl engine-check vllm` on a V100 node once.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
@@ -81,6 +81,10 @@ rather than repeated.
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
|
||||
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
||||
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
@@ -183,6 +187,7 @@ rather than repeated.
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
@@ -645,6 +650,32 @@ called. Empty means vLLM's names.
|
||||
|
||||
`int`
|
||||
|
||||
## EngineOverride
|
||||
|
||||
### `gpu_class`
|
||||
|
||||
`string`
|
||||
|
||||
GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes picks nodes by id.
|
||||
|
||||
### `labels`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Labels picks nodes carrying all of these.
|
||||
|
||||
### `engines`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Engines are laid over the cluster's: a version, or "" for none.
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
@@ -1412,6 +1443,16 @@ never trusted to already have what somebody once put there.
|
||||
Unset keeps DefaultEngines. A version replaces the default; an empty
|
||||
one leaves that engine to be installed by hand.
|
||||
|
||||
### `engine_overrides`
|
||||
|
||||
`[]EngineOverride`
|
||||
|
||||
EngineOverrides give some nodes different engines from the rest: a
|
||||
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
engines over the cluster's, in order, later ones last; an empty version
|
||||
removes that engine from the nodes it picks.
|
||||
|
||||
### `backup_dir`
|
||||
|
||||
`string`
|
||||
|
||||
+48
-3
@@ -104,6 +104,8 @@ type Agent struct {
|
||||
// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
|
||||
// from nvidia-smi.
|
||||
sensors *discovery.SensorSampler
|
||||
// wear reads what wears a GPU out: remapped rows, PCIe retries, XIDs.
|
||||
wear *discovery.HealthSampler
|
||||
|
||||
// fetchWatchers are the shards waiting on each source, so progress on one
|
||||
// download reaches every shard that asked for it. A fetch is single-flight
|
||||
@@ -150,6 +152,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
provKick: make(chan struct{}, 1),
|
||||
load: discovery.NewLoadSampler(),
|
||||
sensors: discovery.NewSensorSampler(),
|
||||
wear: discovery.NewHealthSampler(),
|
||||
}
|
||||
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
|
||||
return a, nil
|
||||
@@ -314,6 +317,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
return
|
||||
}
|
||||
// An engine started by an agent older than the GPU check may be one
|
||||
// serving a GPU placement on the CPU. Stopped, so its replacement starts
|
||||
// through the check and fails there in words, if the build still sees
|
||||
// no GPU — rather than going on unnoticed after the upgrade that could
|
||||
// have caught it.
|
||||
if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
|
||||
a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
_ = a.sup.Stop(ctx, o.ShardID)
|
||||
return
|
||||
}
|
||||
a.log.Info("adopted engine is serving",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID)
|
||||
}
|
||||
@@ -450,7 +464,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
// vLLM accepted a vLLM shard and failed it minutes later.
|
||||
reg.EngineVersions = a.engines(parent)
|
||||
reg.EnginesProbed = true
|
||||
reg.InstallableEngines = provision.Installable()
|
||||
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
|
||||
reg.Features = []string{FeatureFabricTest}
|
||||
reg.RunningShards = a.runningShards()
|
||||
a.mu.Lock()
|
||||
@@ -645,11 +659,34 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
|
||||
if a.sensors != nil {
|
||||
se := a.sensors.Sample(context.Background())
|
||||
out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
|
||||
var wear map[int]discovery.GPUWear
|
||||
if a.wear != nil && len(se.GPUs) > 0 {
|
||||
wear = map[int]discovery.GPUWear{}
|
||||
for _, w := range a.wear.Sample(context.Background()) {
|
||||
wear[w.Index] = w
|
||||
}
|
||||
}
|
||||
for _, g := range se.GPUs {
|
||||
out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
|
||||
gs := &llmgwv1.GPUSensor{
|
||||
Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
|
||||
PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
|
||||
})
|
||||
UtilizationPercent: g.UtilPercent, MemoryUtilizationPercent: g.MemUtilPercent,
|
||||
MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
|
||||
SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
|
||||
}
|
||||
if w, ok := wear[g.Index]; ok {
|
||||
gs.HasWear = true
|
||||
gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
|
||||
w.PCIeReplays, w.RetiredSBE, w.RetiredDBE, w.RetirePending
|
||||
gs.RemappedRowsCorrectable, gs.RemappedRowsUncorrectable, gs.RemapPending, gs.RemapFailure =
|
||||
w.RemappedCorrectable, w.RemappedUncorrectable, w.RemapPending, w.RemapFailure
|
||||
gs.NvlinkCrcErrors, gs.NvlinkReplayErrors, gs.NvlinkRecoveryErrors = w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery
|
||||
gs.XidErrors, gs.LastXid, gs.LastXidText = w.XIDs, int32(w.LastXID), w.LastXIDText
|
||||
if !w.LastXIDAt.IsZero() {
|
||||
gs.LastXidUnix = w.LastXIDAt.Unix()
|
||||
}
|
||||
}
|
||||
out.Gpus = append(out.Gpus, gs)
|
||||
}
|
||||
}
|
||||
return out
|
||||
@@ -931,6 +968,14 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
} else {
|
||||
err = a.sup.WaitAlive(ctx, sh)
|
||||
}
|
||||
if err == nil {
|
||||
err = a.onItsGPUs(ctx, eng, spec, sh.Status().PID)
|
||||
if err != nil {
|
||||
// Serving, but not where it was placed: stopped, so the GPU it
|
||||
// holds in the inventory is not taken by a CPU process.
|
||||
_ = a.sup.Stop(ctx, shardID)
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
report(supervisor.Failed, err.Error(), sh.Logs(20))
|
||||
return
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
func TestAnEngineDisabledOnTheNodeIsNeitherInstalledNorAdvertised(t *testing.T) {
|
||||
a := &Agent{}
|
||||
a.cfg.Engines.Disabled = []string{"vllm"}
|
||||
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{
|
||||
{Engine: "llamacpp", Version: "b1"}, {Engine: "vllm", Version: "0.30.0"},
|
||||
}})
|
||||
if len(a.provWants) != 1 || a.provWants[0].Engine != "llamacpp" {
|
||||
t.Errorf("wants %+v: vLLM is disabled here and must not be installed", a.provWants)
|
||||
}
|
||||
a.configuredVers = map[string]string{"vllm": "0.25.0"}
|
||||
a.installedVers = map[string]string{"llamacpp": "b1"}
|
||||
if got := a.engineReportLocked(); got["vllm"] != "" || got["llamacpp"] != "b1" {
|
||||
t.Errorf("advertised %v: a disabled engine found on disk is still not offered", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineNamedByHandIsNotInstalledOverIt(t *testing.T) {
|
||||
a := &Agent{reg: engines.NewRegistry(engines.Config{LlamaCpp: engines.LlamaCppConfig{Bin: "/opt/own/llama-server"}})}
|
||||
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "llamacpp", Version: "b1"}}})
|
||||
if len(a.provWants) != 0 {
|
||||
t.Errorf("wants %+v: an installed build would be used over the operator's own", a.provWants)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// GPUWear is what wears a card out or takes it down, beyond temperature:
|
||||
// read every minute, from nvidia-smi and the kernel's log. The early signs of
|
||||
// a card failing are here — a growing count of rows its memory had to remap,
|
||||
// PCIe link retries, XID errors — long before a replica on it fails.
|
||||
//
|
||||
// A count of -1 is not reported: a card without row remapping (before
|
||||
// Ampere) has no such count, which is not the same as none.
|
||||
type GPUWear struct {
|
||||
Index int
|
||||
// PCIeReplays are link-level retries on the card's PCIe link: a few are
|
||||
// noise, a count that keeps climbing is a riser, slot or cable.
|
||||
PCIeReplays int64
|
||||
// RetiredSBE and RetiredDBE are memory pages retired for single and
|
||||
// double bit errors (cards before Ampere); RetirePending is a page
|
||||
// waiting for a reset to be retired.
|
||||
RetiredSBE, RetiredDBE int64
|
||||
RetirePending bool
|
||||
// RemappedCorrectable and RemappedUncorrectable are memory rows remapped
|
||||
// (Ampere and later). RemapPending waits for a reset; RemapFailure is a
|
||||
// card out of spare rows — it needs replacing.
|
||||
RemappedCorrectable, RemappedUncorrectable int64
|
||||
RemapPending, RemapFailure bool
|
||||
// NVLink errors summed over the card's links, by kind.
|
||||
NVLinkCRC, NVLinkReplay, NVLinkRecovery int64
|
||||
// XIDs are the driver's error events for this card since the agent
|
||||
// started, and the last of them.
|
||||
XIDs int64
|
||||
LastXID int
|
||||
LastXIDText string
|
||||
LastXIDAt time.Time
|
||||
}
|
||||
|
||||
// healthEvery is how often health is read: it changes over hours, and the
|
||||
// kernel log and nvidia-smi's full report are the heavier reads.
|
||||
const healthEvery = time.Minute
|
||||
|
||||
// HealthSampler reads GPUWear, keeping the XID counts between reads.
|
||||
type HealthSampler struct {
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
at time.Time
|
||||
last []GPUWear
|
||||
since time.Time
|
||||
xids map[int]*GPUWear
|
||||
started bool
|
||||
}
|
||||
|
||||
// NewHealthSampler reads the machine's own nvidia-smi and kernel log.
|
||||
func NewHealthSampler() *HealthSampler {
|
||||
return &HealthSampler{now: time.Now, xids: map[int]*GPUWear{},
|
||||
run: func(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
return exec.CommandContext(ctx, name, args...).Output() //nolint:gosec // fixed tools
|
||||
}}
|
||||
}
|
||||
|
||||
// Sample is the cards' health, read again once healthEvery has passed. Nil
|
||||
// on a machine without nvidia-smi.
|
||||
func (h *HealthSampler) Sample(ctx context.Context) []GPUWear {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
now := h.now()
|
||||
if !h.at.IsZero() && now.Sub(h.at) < healthEvery {
|
||||
return h.last
|
||||
}
|
||||
h.at = now
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
report, err := h.run(ctx, "nvidia-smi", "-q", "-x")
|
||||
if err != nil {
|
||||
h.last = nil
|
||||
return nil
|
||||
}
|
||||
cards, bus := parseHealthReport(report)
|
||||
if out, err := h.run(ctx, "nvidia-smi", "nvlink", "-e"); err == nil {
|
||||
for i, e := range parseNVLinkErrors(string(out)) {
|
||||
if i < len(cards) {
|
||||
cards[i].NVLinkCRC, cards[i].NVLinkReplay, cards[i].NVLinkRecovery = e[0], e[1], e[2]
|
||||
}
|
||||
}
|
||||
}
|
||||
h.readXIDs(ctx, now, bus)
|
||||
for i := range cards {
|
||||
if x := h.xids[i]; x != nil {
|
||||
cards[i].XIDs, cards[i].LastXID, cards[i].LastXIDText, cards[i].LastXIDAt = x.XIDs, x.LastXID, x.LastXIDText, x.LastXIDAt
|
||||
}
|
||||
}
|
||||
h.last = cards
|
||||
return cards
|
||||
}
|
||||
|
||||
// readXIDs adds the XID events logged since the last read. The first read
|
||||
// starts from now: what happened before the agent started was before, and
|
||||
// counting the whole log would raise every alert on every restart.
|
||||
func (h *HealthSampler) readXIDs(ctx context.Context, now time.Time, bus map[string]int) {
|
||||
if !h.started {
|
||||
h.started, h.since = true, now
|
||||
return
|
||||
}
|
||||
since := h.since
|
||||
h.since = now
|
||||
out, err := h.run(ctx, "journalctl", "-k", "--no-pager", "-o", "short-unix",
|
||||
"--since", "@"+strconv.FormatInt(since.Unix(), 10), "-g", "NVRM: Xid")
|
||||
if err != nil {
|
||||
// No journal, or nothing matched (journalctl exits 1 for that).
|
||||
return
|
||||
}
|
||||
for _, e := range parseXIDs(string(out)) {
|
||||
i, ok := bus[e.bus]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
x := h.xids[i]
|
||||
if x == nil {
|
||||
x = &GPUWear{Index: i}
|
||||
h.xids[i] = x
|
||||
}
|
||||
x.XIDs++
|
||||
x.LastXID, x.LastXIDText = e.code, e.text
|
||||
x.LastXIDAt = now
|
||||
if e.at > 0 {
|
||||
x.LastXIDAt = time.Unix(int64(e.at), 0)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// healthLog is the part of nvidia-smi's XML report health is read from.
|
||||
type healthLog struct {
|
||||
GPUs []struct {
|
||||
ID string `xml:"id,attr"`
|
||||
PCI struct {
|
||||
Replay string `xml:"replay_counter"`
|
||||
} `xml:"pci"`
|
||||
Retired struct {
|
||||
SBE string `xml:"multiple_single_bit_retirement>retired_count"`
|
||||
DBE string `xml:"double_bit_retirement>retired_count"`
|
||||
Pending string `xml:"pending_retirement"`
|
||||
// Older drivers name it pending_blacklist.
|
||||
Blacklist string `xml:"pending_blacklist"`
|
||||
} `xml:"retired_pages"`
|
||||
Remapped struct {
|
||||
Corr string `xml:"remapped_row_corr"`
|
||||
Unc string `xml:"remapped_row_unc"`
|
||||
Pending string `xml:"remapped_row_pending"`
|
||||
Failure string `xml:"remapped_row_failure"`
|
||||
} `xml:"remapped_rows"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseHealthReport reads each card's health, in index order, and maps each
|
||||
// card's PCI bus ("3b:00", as the kernel names it) to its index.
|
||||
func parseHealthReport(report []byte) ([]GPUWear, map[string]int) {
|
||||
var log healthLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
bus := map[string]int{}
|
||||
out := make([]GPUWear, 0, len(log.GPUs))
|
||||
for i, g := range log.GPUs {
|
||||
bus[busOf(g.ID)] = i
|
||||
out = append(out, GPUWear{
|
||||
Index: i,
|
||||
PCIeReplays: count(g.PCI.Replay),
|
||||
RetiredSBE: count(g.Retired.SBE),
|
||||
RetiredDBE: count(g.Retired.DBE),
|
||||
RetirePending: yes(g.Retired.Pending) || yes(g.Retired.Blacklist),
|
||||
RemappedCorrectable: count(g.Remapped.Corr),
|
||||
RemappedUncorrectable: count(g.Remapped.Unc),
|
||||
RemapPending: yes(g.Remapped.Pending),
|
||||
RemapFailure: yes(g.Remapped.Failure),
|
||||
})
|
||||
}
|
||||
return out, bus
|
||||
}
|
||||
|
||||
// busOf is a PCI address as the kernel's XID lines carry it: nvidia-smi
|
||||
// says "00000000:3B:00.0", the kernel "0000:3b:00".
|
||||
func busOf(id string) string {
|
||||
id = strings.ToLower(strings.TrimSpace(id))
|
||||
if i := strings.LastIndex(id, "."); i >= 0 {
|
||||
id = id[:i]
|
||||
}
|
||||
parts := strings.Split(id, ":")
|
||||
if len(parts) >= 2 {
|
||||
return parts[len(parts)-2] + ":" + parts[len(parts)-1]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func count(s string) int64 {
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 64)
|
||||
if err != nil {
|
||||
return -1
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func yes(s string) bool { return strings.EqualFold(strings.TrimSpace(s), "yes") }
|
||||
|
||||
// nvlinkError matches a line of `nvidia-smi nvlink -e`, whose wording varies
|
||||
// by driver: "Link 0: Replay Errors: 0", "Link 0: CRC Errors: 0",
|
||||
// "Link 3: Data CRC Errors: 2".
|
||||
var (
|
||||
nvlinkGPU = regexp.MustCompile(`^GPU (\d+):`)
|
||||
nvlinkError = regexp.MustCompile(`Link \d+:\s*(.*?)\s*Errors?\s*:\s*(\d+)`)
|
||||
)
|
||||
|
||||
// parseNVLinkErrors sums each card's link errors by kind: CRC, replay,
|
||||
// recovery.
|
||||
func parseNVLinkErrors(out string) map[int][3]int64 {
|
||||
res := map[int][3]int64{}
|
||||
gpu := -1
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if m := nvlinkGPU.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
|
||||
gpu, _ = strconv.Atoi(m[1])
|
||||
continue
|
||||
}
|
||||
m := nvlinkError.FindStringSubmatch(line)
|
||||
if m == nil || gpu < 0 {
|
||||
continue
|
||||
}
|
||||
n, _ := strconv.ParseInt(m[2], 10, 64)
|
||||
e := res[gpu]
|
||||
kind := strings.ToLower(m[1])
|
||||
switch {
|
||||
case strings.Contains(kind, "crc"):
|
||||
e[0] += n
|
||||
case strings.Contains(kind, "replay"):
|
||||
e[1] += n
|
||||
case strings.Contains(kind, "recovery"):
|
||||
e[2] += n
|
||||
}
|
||||
res[gpu] = e
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
type xidEvent struct {
|
||||
bus string
|
||||
code int
|
||||
text string
|
||||
at float64
|
||||
}
|
||||
|
||||
// xidLine matches the driver's report of an XID:
|
||||
// "NVRM: Xid (PCI:0000:3b:00): 79, pid=..., GPU has fallen off the bus."
|
||||
var xidLine = regexp.MustCompile(`NVRM: Xid \(PCI:([0-9a-fA-F:.]+)\): (\d+),\s*(.*)$`)
|
||||
|
||||
// parseXIDs reads journalctl's short-unix lines: a timestamp, then the
|
||||
// message.
|
||||
func parseXIDs(out string) []xidEvent {
|
||||
var evs []xidEvent
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
m := xidLine.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
code, _ := strconv.Atoi(m[2])
|
||||
e := xidEvent{bus: busOf(m[1]), code: code, text: strings.TrimSpace(m[3])}
|
||||
if f := strings.Fields(line); len(f) > 0 {
|
||||
e.at, _ = strconv.ParseFloat(f[0], 64)
|
||||
}
|
||||
evs = append(evs, e)
|
||||
}
|
||||
return evs
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const healthXML = `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<gpu id="00000000:3B:00.0">
|
||||
<pci><replay_counter>0</replay_counter></pci>
|
||||
<retired_pages>
|
||||
<multiple_single_bit_retirement><retired_count>N/A</retired_count></multiple_single_bit_retirement>
|
||||
<double_bit_retirement><retired_count>N/A</retired_count></double_bit_retirement>
|
||||
<pending_retirement>N/A</pending_retirement>
|
||||
</retired_pages>
|
||||
<remapped_rows>
|
||||
<remapped_row_corr>2</remapped_row_corr>
|
||||
<remapped_row_unc>1</remapped_row_unc>
|
||||
<remapped_row_pending>Yes</remapped_row_pending>
|
||||
<remapped_row_failure>No</remapped_row_failure>
|
||||
</remapped_rows>
|
||||
</gpu>
|
||||
<gpu id="00000000:5E:00.0">
|
||||
<pci><replay_counter>4021</replay_counter></pci>
|
||||
<retired_pages>
|
||||
<multiple_single_bit_retirement><retired_count>3</retired_count></multiple_single_bit_retirement>
|
||||
<double_bit_retirement><retired_count>1</retired_count></double_bit_retirement>
|
||||
<pending_blacklist>Yes</pending_blacklist>
|
||||
</retired_pages>
|
||||
<remapped_rows>N/A</remapped_rows>
|
||||
</gpu>
|
||||
</nvidia_smi_log>`
|
||||
|
||||
const nvlinkOut = `GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-aaa)
|
||||
Link 0: Replay Errors: 1
|
||||
Link 0: Recovery Errors: 0
|
||||
Link 0: CRC Errors: 5
|
||||
Link 1: Data CRC Errors: 2
|
||||
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-bbb)
|
||||
Link 0: Replay Errors: 0
|
||||
`
|
||||
|
||||
func TestAGPUsWearAndFailuresAreReadAndCountedFromWhenTheAgentStarted(t *testing.T) {
|
||||
now := time.Unix(1_800_000_000, 0)
|
||||
journal := ""
|
||||
h := &HealthSampler{now: func() time.Time { return now }, xids: map[int]*GPUWear{},
|
||||
run: func(_ context.Context, name string, args ...string) ([]byte, error) {
|
||||
switch {
|
||||
case name == "nvidia-smi" && args[0] == "-q":
|
||||
return []byte(healthXML), nil
|
||||
case name == "nvidia-smi" && args[0] == "nvlink":
|
||||
return []byte(nvlinkOut), nil
|
||||
case name == "journalctl":
|
||||
if journal == "" {
|
||||
return nil, errors.New("exit status 1")
|
||||
}
|
||||
return []byte(journal), nil
|
||||
}
|
||||
return nil, errors.New("unexpected " + name)
|
||||
}}
|
||||
got := h.Sample(t.Context())
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
a, b := got[0], got[1]
|
||||
if a.RemappedCorrectable != 2 || a.RemappedUncorrectable != 1 || !a.RemapPending || a.RemapFailure ||
|
||||
a.RetiredSBE != -1 || a.PCIeReplays != 0 || a.NVLinkCRC != 7 || a.NVLinkReplay != 1 {
|
||||
t.Errorf("card 0 %+v", a)
|
||||
}
|
||||
if b.PCIeReplays != 4021 || b.RetiredSBE != 3 || b.RetiredDBE != 1 || !b.RetirePending || b.RemappedCorrectable != -1 {
|
||||
t.Errorf("card 1 %+v", b)
|
||||
}
|
||||
|
||||
// An XID on card 1, logged after the agent started; read on the next
|
||||
// sample a minute on.
|
||||
journal = "1800000030.123456 gpu07 kernel: NVRM: Xid (PCI:0000:5e:00): 79, pid=1234, name=llama-server, GPU has fallen off the bus.\n"
|
||||
now = now.Add(61 * time.Second)
|
||||
got = h.Sample(t.Context())
|
||||
if got[1].XIDs != 1 || got[1].LastXID != 79 || !strings.Contains(got[1].LastXIDText, "fallen off the bus") ||
|
||||
got[1].LastXIDAt.Unix() != 1_800_000_030 || got[0].XIDs != 0 {
|
||||
t.Errorf("after an XID: %+v / %+v", got[0], got[1])
|
||||
}
|
||||
// Within the minute, the last reading.
|
||||
journal = ""
|
||||
now = now.Add(10 * time.Second)
|
||||
if again := h.Sample(t.Context()); again[1].XIDs != 1 {
|
||||
t.Errorf("%+v", again[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestBusIDsAreReadAsTheKernelWritesThem(t *testing.T) {
|
||||
for in, want := range map[string]string{"00000000:3B:00.0": "3b:00", "0000:3b:00": "3b:00", "0000:5e:00.0": "5e:00"} {
|
||||
if got := busOf(in); got != want {
|
||||
t.Errorf("%s: %s, want %s", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
|
||||
// nvidia-smi index: what a process actually computes on, as opposed to what it
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
|
||||
// backend did not load runs on the CPU without a word, and this is where that
|
||||
// shows — the process is on no GPU's list.
|
||||
//
|
||||
// From the full query rather than --query-compute-apps, which lists compute
|
||||
// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
|
||||
// would have read as not on its GPU at all.
|
||||
//
|
||||
// A node with no nvidia-smi returns nil and no error, as discovery does.
|
||||
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
if binary == "" {
|
||||
if _, err := exec.LookPath("nvidia-smi"); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
binary = "nvidia-smi"
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseGPUProcesses(out)
|
||||
}
|
||||
|
||||
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
|
||||
// index order, and the processes on it.
|
||||
type smiLog struct {
|
||||
GPUs []struct {
|
||||
Processes []struct {
|
||||
PID string `xml:"pid"`
|
||||
Type string `xml:"type"`
|
||||
} `xml:"processes>process_info"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseGPUProcesses reads the report. GPUs are listed in index order, which
|
||||
// is nvidia-smi's own numbering — the one placement assigns by.
|
||||
func parseGPUProcesses(report []byte) (map[int][]int, error) {
|
||||
var log smiLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
out := map[int][]int{}
|
||||
for i, g := range log.GPUs {
|
||||
for _, p := range g.Processes {
|
||||
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
|
||||
out[i] = append(out[i], pid)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
|
||||
report := `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<driver_version>580.65.06</driver_version>
|
||||
<gpu id="00000000:17:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes></processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:65:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes>
|
||||
<process_info><pid>4100</pid><type>C</type><process_name>llama-server</process_name></process_info>
|
||||
<process_info><pid>4300</pid><type>G</type><process_name>llama-server</process_name></process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:B3:00.0"><processes>
|
||||
<process_info><pid>4200</pid><type>C+G</type></process_info>
|
||||
</processes></gpu>
|
||||
</nvidia_smi_log>`
|
||||
got, err := parseGPUProcesses([]byte(report))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A Vulkan engine is a graphics process ("G") and is on its GPU all the
|
||||
// same; a compute-only query would have missed it.
|
||||
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
|
||||
t.Error("a report that does not parse was read as no processes")
|
||||
}
|
||||
}
|
||||
@@ -41,6 +41,14 @@ type GPUSensor struct {
|
||||
// is running free. Idle and application clock settings are left out;
|
||||
// they are not the card being held back.
|
||||
Throttle []string
|
||||
// How busy it is, read with the rest every sample. Discovery reads the
|
||||
// same once, at start, which made a busy card look as idle as it was
|
||||
// when the agent came up. -1 is not reported by the driver; zero
|
||||
// MemoryTotalMB is a sample without these at all.
|
||||
UtilPercent, MemUtilPercent float64
|
||||
MemoryUsedMB, MemoryTotalMB float64
|
||||
SMClockMHz float64
|
||||
ECCUncorrected float64
|
||||
}
|
||||
|
||||
// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
|
||||
@@ -187,13 +195,33 @@ var throttleBits = []struct {
|
||||
// gpus asks nvidia-smi for each card's heat and power. The memory temperature
|
||||
// is not known to every driver, and a field nvidia-smi does not know fails the
|
||||
// whole query, so it is asked for first and dropped on failure.
|
||||
// gpuFields are what each sample asks nvidia-smi for, in order. The busy
|
||||
// figures follow the throttle reasons; the memory temperature is last, being
|
||||
// the one field a driver may reject outright.
|
||||
const (
|
||||
gpuBase = "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active"
|
||||
gpuBusy = ",utilization.gpu,utilization.memory,memory.used,memory.total,clocks.sm,ecc.errors.uncorrected.volatile.total"
|
||||
gpuHBMTC = ",temperature.memory"
|
||||
)
|
||||
|
||||
func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
|
||||
withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
|
||||
out, err := s.smi(ctx, withMem)
|
||||
if err != nil {
|
||||
if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
|
||||
return nil
|
||||
// Most fields first, fewer on a driver that refuses one: a query naming
|
||||
// a field the driver does not know fails as a whole.
|
||||
var out []byte
|
||||
var busy bool
|
||||
for _, q := range []struct {
|
||||
fields string
|
||||
busy bool
|
||||
}{{gpuBase + gpuBusy + gpuHBMTC, true}, {gpuBase + gpuBusy, true}, {gpuBase + gpuHBMTC, false}, {gpuBase, false}} {
|
||||
var err error
|
||||
if out, err = s.smi(ctx, q.fields); err == nil {
|
||||
busy = q.busy
|
||||
break
|
||||
}
|
||||
out = nil
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
var gs []GPUSensor
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
@@ -216,14 +244,31 @@ func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(f) > 5 {
|
||||
g.MemoryTempC = num(f[5])
|
||||
rest := f[5:]
|
||||
if busy && len(rest) >= 6 {
|
||||
g.UtilPercent, g.MemUtilPercent = numOrUnknown(rest[0]), numOrUnknown(rest[1])
|
||||
g.MemoryUsedMB, g.MemoryTotalMB = num(rest[2]), num(rest[3])
|
||||
g.SMClockMHz, g.ECCUncorrected = num(rest[4]), numOrUnknown(rest[5])
|
||||
rest = rest[6:]
|
||||
}
|
||||
if len(rest) > 0 {
|
||||
g.MemoryTempC = num(rest[0])
|
||||
}
|
||||
gs = append(gs, g)
|
||||
}
|
||||
return gs
|
||||
}
|
||||
|
||||
// numOrUnknown is a figure, or -1 where nvidia-smi says it has none: a card
|
||||
// without ECC has no error count, which is not the same as none.
|
||||
func numOrUnknown(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil || v < 0 {
|
||||
return -1
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
|
||||
func num(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
|
||||
@@ -167,3 +167,44 @@ func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T
|
||||
t.Errorf("%+v", got.GPUs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachSampleSaysHowBusyEachGPUIs(t *testing.T) {
|
||||
// What the card is doing now, every sample: discovery read it once, at
|
||||
// start, and a card busy since looked as idle as it was then.
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
if !strings.Contains(fields, "utilization.gpu") {
|
||||
return nil, errors.New("asked without how busy the cards are")
|
||||
}
|
||||
return []byte(
|
||||
"0, 71, 612.45, 700.00, 0x0000000000000000, 97, 64, 71234, 81559, 1980, 0, 84\n" +
|
||||
"1, 30, 70.10, 700.00, 0x0000000000000001, 0, 0, 512, 81559, 345, [N/A], [N/A]\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
g0, g1 := got[0], got[1]
|
||||
if g0.UtilPercent != 97 || g0.MemUtilPercent != 64 || g0.MemoryUsedMB != 71234 || g0.MemoryTotalMB != 81559 ||
|
||||
g0.SMClockMHz != 1980 || g0.ECCUncorrected != 0 || g0.MemoryTempC != 84 {
|
||||
t.Errorf("card 0 %+v", g0)
|
||||
}
|
||||
// Idle is zero; a count the card does not keep is unknown, not zero.
|
||||
if g1.UtilPercent != 0 || g1.ECCUncorrected != -1 || g1.MemoryTempC != 0 {
|
||||
t.Errorf("card 1 %+v", g1)
|
||||
}
|
||||
}
|
||||
|
||||
func TestADriverThatDoesNotKnowTheBusyFieldsStillReportsHeatAndPower(t *testing.T) {
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
if strings.Contains(fields, "utilization.gpu") {
|
||||
return nil, errors.New(`Field "ecc.errors.uncorrected.volatile.total" is not a valid field to query.`)
|
||||
}
|
||||
return []byte("0, 66, 250.0, 300.0, 0x0000000000000000, 70\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
|
||||
if len(got) != 1 || got[0].TempC != 66 || got[0].MemoryTempC != 70 || got[0].MemoryTotalMB != 0 {
|
||||
t.Errorf("%+v", got)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"slices"
|
||||
@@ -146,6 +147,15 @@ type Engine interface {
|
||||
MetricsPath() string
|
||||
}
|
||||
|
||||
// CPUFallback is an engine that can come up on the CPU when it was placed on
|
||||
// GPUs, and keep serving there without an error. The agent checks such a shard
|
||||
// is on its GPUs before calling it ready.
|
||||
type CPUFallback interface {
|
||||
FallsBackToCPU() bool
|
||||
// Devices lists the GPUs the engine's build can see, for saying why.
|
||||
Devices(ctx context.Context) ([]string, error)
|
||||
}
|
||||
|
||||
// GPUInfo is the subset of GPU facts validation needs.
|
||||
type GPUInfo struct {
|
||||
Index int
|
||||
@@ -203,6 +213,10 @@ func (r *Registry) Names() []string {
|
||||
|
||||
// Config configures the engine adapters.
|
||||
type Config struct {
|
||||
// Disabled are engines this node never installs or advertises, whatever
|
||||
// the control plane asks: vLLM on a node of cards no build of it runs on,
|
||||
// that is meant for llama.cpp alone.
|
||||
Disabled []string `koanf:"disabled"`
|
||||
VLLM VLLMConfig `koanf:"vllm"`
|
||||
SGLang SGLangConfig `koanf:"sglang"`
|
||||
LlamaCpp LlamaCppConfig `koanf:"llamacpp"`
|
||||
@@ -232,6 +246,11 @@ func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
|
||||
func baseEnv(s ShardSpec) map[string]string {
|
||||
env := map[string]string{
|
||||
"CUDA_VISIBLE_DEVICES": joinInts(s.GPUIndices),
|
||||
// The indices above are nvidia-smi's, which counts by PCI bus. CUDA
|
||||
// counts fastest first unless told otherwise, and on a node with two
|
||||
// kinds of GPU the shard would get a different card than placement
|
||||
// gave it — one another shard may hold.
|
||||
"CUDA_DEVICE_ORDER": "PCI_BUS_ID",
|
||||
}
|
||||
|
||||
if len(s.HCAs) > 0 {
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sort"
|
||||
@@ -65,25 +67,25 @@ func (*LlamaCpp) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// Validate checks the profile against the GPUs it would run on.
|
||||
//
|
||||
// The two refusals here are about not pretending: llama.cpp has no equivalent
|
||||
// of a tensor-parallel group spanning machines, and its multi-GPU support is a
|
||||
// different mechanism (--split-mode, --tensor-split) that this adapter does not
|
||||
// drive. Accepting either would produce a replica that starts and then serves
|
||||
// from one machine while the catalog believes it has several.
|
||||
// llama.cpp spreads a model over a node's GPUs by layer (--split-mode layer),
|
||||
// so gpus_per_node is how many it gets. It has no tensor parallelism and no
|
||||
// group spanning machines, and accepting tp or nodes would produce a replica
|
||||
// that serves from fewer cards, or one machine, while the catalog believes
|
||||
// otherwise.
|
||||
func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if err := validateCommon(p, gpus); err != nil {
|
||||
return err
|
||||
}
|
||||
if p.Nodes > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
p.Name, p.Nodes)
|
||||
}
|
||||
if p.TP > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp does not do tensor parallelism; "+
|
||||
"it splits a model across local GPUs with --tensor-split, which this adapter does not drive",
|
||||
p.Name)
|
||||
if p.TP > 1 || p.PP > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp splits a model across a node's GPUs by layer, not by tp or pp; "+
|
||||
"write gpus_per_node: %d instead", p.Name, max(p.TP, 1)*max(p.PP, 1))
|
||||
}
|
||||
return nil
|
||||
// The common checks count a replica's GPUs as tp; for llama.cpp the
|
||||
// count is gpus_per_node.
|
||||
q := p
|
||||
q.TP = max(p.GPUsPerNode, 1)
|
||||
return validateCommon(q, gpus)
|
||||
}
|
||||
|
||||
// Commands returns the processes for one shard, which is always one process.
|
||||
@@ -132,6 +134,13 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
|
||||
args = append(args, "--n-gpu-layers", "999")
|
||||
}
|
||||
// Several GPUs: the layers spread across all of them, which is the only
|
||||
// split llama.cpp does without a matched interconnect. In proportion to
|
||||
// each card's free memory, which for the whole cards placement gives is
|
||||
// an even split.
|
||||
if len(s.GPUIndices) > 1 && !hasFlag(p.Args, "--split-mode", "-sm") {
|
||||
args = append(args, "--split-mode", "layer")
|
||||
}
|
||||
if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
|
||||
// What placement reserved first: it is what the node's other
|
||||
// replicas were placed around.
|
||||
@@ -159,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if s.APIKey != "" {
|
||||
env["LLAMA_API_KEY"] = s.APIKey
|
||||
}
|
||||
// The Vulkan build — what a GPU older than the CUDA builds gets — does
|
||||
// not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
|
||||
// as the driver lists them, which on NVIDIA follows the same bus order
|
||||
// as nvidia-smi; were it not to, the shard would not hold memory on its
|
||||
// own GPUs, and starting it checks exactly that.
|
||||
if len(s.GPUIndices) > 0 {
|
||||
env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
|
||||
}
|
||||
return []Command{{
|
||||
Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
|
||||
}}, nil
|
||||
@@ -234,3 +251,38 @@ func (l *LlamaCpp) bin() string {
|
||||
}
|
||||
return "llama-server"
|
||||
}
|
||||
|
||||
// FallsBackToCPU says that llama.cpp keeps running when its GPU backend does
|
||||
// not load: --n-gpu-layers asks for layers on a GPU it then has none of, and
|
||||
// the server comes up and answers on the CPU, at a tenth of the speed, with no
|
||||
// error anywhere. Its readiness on a GPU placement needs checking separately.
|
||||
func (*LlamaCpp) FallsBackToCPU() bool { return true }
|
||||
|
||||
// Devices lists the GPUs this build can compute on, as llama-server's
|
||||
// --list-devices names them ("CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)").
|
||||
// Empty, on a machine with GPUs, is a build whose GPU backend did not load:
|
||||
// the CPU build, or a CUDA build missing a library its backend links.
|
||||
func (l *LlamaCpp) Devices(ctx context.Context) ([]string, error) {
|
||||
out, err := exec.CommandContext(ctx, l.bin(), "--list-devices").CombinedOutput() //nolint:gosec // the engine's own binary
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s --list-devices: %v: %s", l.bin(), err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return parseDevices(string(out)), nil
|
||||
}
|
||||
|
||||
// parseDevices reads --list-devices: the lines after "Available devices:",
|
||||
// which says "(none)" when there are none. Log lines before it are skipped.
|
||||
func parseDevices(out string) []string {
|
||||
var devs []string
|
||||
listing := false
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
switch {
|
||||
case strings.HasPrefix(line, "Available devices"):
|
||||
listing = true
|
||||
case listing && line != "" && line != "(none)":
|
||||
devs = append(devs, line)
|
||||
}
|
||||
}
|
||||
return devs
|
||||
}
|
||||
@@ -323,3 +323,41 @@ func TestAdaptersAreLoadedAtStartAndAppliedToNoneByDefault(t *testing.T) {
|
||||
t.Error("a model with no adapters was given lora flags")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDevicesAreReadFromListDevices(t *testing.T) {
|
||||
cuda := "ggml_cuda_init: found 2 CUDA devices:\nAvailable devices:\n CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n CUDA1: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
|
||||
if got := parseDevices(cuda); len(got) != 2 || !strings.HasPrefix(got[0], "CUDA0: NVIDIA L4") {
|
||||
t.Errorf("CUDA build: %q", got)
|
||||
}
|
||||
// A CUDA build whose backend did not load says what the CPU build says.
|
||||
if got := parseDevices("Available devices:\n (none)\n"); len(got) != 0 {
|
||||
t.Errorf("no devices: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLlamaCppSpreadsAReplicaOverItsNodesGPUsByLayer(t *testing.T) {
|
||||
four := Profile{Name: "t4x4", Engine: "llamacpp", GPUClass: "t4", TP: 1, GPUsPerNode: 4, ModelPath: "/m"}
|
||||
if err := (&LlamaCpp{}).Validate(four, h100(4)); err != nil {
|
||||
t.Fatalf("four GPUs on a four-GPU node: %v", err)
|
||||
}
|
||||
if err := (&LlamaCpp{}).Validate(four, h100(2)); err == nil {
|
||||
t.Error("four GPUs were accepted on a node with two")
|
||||
}
|
||||
|
||||
s := llamaSpec([]int{0, 1, 2, 3})
|
||||
s.Profile.GPUsPerNode = 4
|
||||
cmds, err := (&LlamaCpp{}).Commands(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, _ := flagValue(cmds[0].Args, "--split-mode"); v != "layer" {
|
||||
t.Errorf("--split-mode %q over four GPUs", v)
|
||||
}
|
||||
if !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0,1,2,3") {
|
||||
t.Errorf("env %v: the replica must see all four of its GPUs", cmds[0].Env)
|
||||
}
|
||||
one, _ := (&LlamaCpp{}).Commands(llamaSpec([]int{2}))
|
||||
if hasFlag(one[0].Args, "--split-mode") {
|
||||
t.Error("a one-GPU replica was given a split mode")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
// gpuProcesses and processGroup are variables so tests can stand in for
|
||||
// nvidia-smi and /proc.
|
||||
var (
|
||||
gpuProcesses = discovery.GPUProcesses
|
||||
processGroup = pgidOf
|
||||
// gpuCheckWait is how long a shard has, once it answers, to show up on
|
||||
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
|
||||
gpuCheckWait = 15 * time.Second
|
||||
)
|
||||
|
||||
// onItsGPUs checks that a shard placed on GPUs computes on them.
|
||||
//
|
||||
// For an engine that falls back to the CPU when its GPU backend does not load,
|
||||
// "it answers" is not "it is ready": llama.cpp built for CUDA but missing
|
||||
// cuBLAS comes up on the CPU, answers at a tenth of the speed, and the GPU it
|
||||
// was given sits idle and reserved. What it computes on is what nvidia-smi
|
||||
// lists it on, so a shard none of whose processes hold memory on one of its
|
||||
// GPUs is refused, with what the engine itself can see.
|
||||
func (a *Agent) onItsGPUs(ctx context.Context, eng engines.Engine, spec engines.ShardSpec, pid int) error {
|
||||
fb, ok := eng.(engines.CPUFallback)
|
||||
if !ok || !fb.FallsBackToCPU() || len(spec.GPUIndices) == 0 || pid <= 0 {
|
||||
return nil
|
||||
}
|
||||
group := processGroup(pid)
|
||||
deadline := time.Now().Add(gpuCheckWait)
|
||||
var missing []int
|
||||
for {
|
||||
procs, err := gpuProcesses(ctx, a.cfg.Discovery.NvidiaSMI)
|
||||
if err != nil || procs == nil {
|
||||
// No nvidia-smi to ask is no evidence either way, and a node that
|
||||
// was given GPUs has one; say so and let the shard serve.
|
||||
a.log.Warn("could not check that a GPU shard runs on its GPUs", "shard", spec.ShardID, "err", err)
|
||||
return nil
|
||||
}
|
||||
missing = missing[:0]
|
||||
for _, i := range spec.GPUIndices {
|
||||
if !holds(procs[i], pid, group) {
|
||||
missing = append(missing, i)
|
||||
}
|
||||
}
|
||||
if len(missing) == 0 {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
seen := "it lists no GPU at all"
|
||||
if devs, err := fb.Devices(ctx); err != nil {
|
||||
seen = "listing its devices failed: " + err.Error()
|
||||
} else if len(devs) > 0 {
|
||||
seen = "it lists " + strings.Join(devs, "; ")
|
||||
}
|
||||
return fmt.Errorf("placed on GPU %s but computing on the CPU: the engine holds no memory there. "+
|
||||
"Its GPU backend did not load (%s). Check the build with `ldd` on its libggml-cuda.so, "+
|
||||
"or run `llmgw doctor` on this node", joinIndices(missing), seen)
|
||||
}
|
||||
|
||||
// holds reports whether any of pids is the shard's process or in its group:
|
||||
// an engine that computes in a child process is still the shard.
|
||||
func holds(pids []int, pid, group int) bool {
|
||||
for _, p := range pids {
|
||||
if p == pid || (group > 0 && processGroup(p) == group) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// pgidOf reads a process's group from /proc; zero when it cannot.
|
||||
func pgidOf(pid int) int {
|
||||
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
// The command name, in parentheses, may hold spaces; the fields after its
|
||||
// closing parenthesis are state, ppid, pgrp.
|
||||
s := string(b)
|
||||
i := strings.LastIndexByte(s, ')')
|
||||
if i < 0 {
|
||||
return 0
|
||||
}
|
||||
f := strings.Fields(s[i+1:])
|
||||
if len(f) < 3 {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.Atoi(f[2])
|
||||
return n
|
||||
}
|
||||
|
||||
func joinIndices(is []int) string {
|
||||
s := make([]string, len(is))
|
||||
for k, i := range is {
|
||||
s[k] = strconv.Itoa(i)
|
||||
}
|
||||
return strings.Join(s, ", ")
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
// fallbackEngine is an engine that comes up on the CPU when its GPU backend
|
||||
// does not load, and lists the devices it can see.
|
||||
type fallbackEngine struct {
|
||||
engines.Engine
|
||||
devices []string
|
||||
}
|
||||
|
||||
func (fallbackEngine) FallsBackToCPU() bool { return true }
|
||||
func (f fallbackEngine) Devices(context.Context) ([]string, error) {
|
||||
return f.devices, nil
|
||||
}
|
||||
|
||||
func TestAGPUShardComputingOnTheCPUIsNotReady(t *testing.T) {
|
||||
procs := map[int][]int{}
|
||||
oldProcs, oldGroup, oldWait := gpuProcesses, processGroup, gpuCheckWait
|
||||
gpuProcesses = func(context.Context, string) (map[int][]int, error) { return procs, nil }
|
||||
processGroup = func(pid int) int {
|
||||
if pid >= 500 && pid < 600 {
|
||||
return 500 // the shard's group: 500 and its children
|
||||
}
|
||||
return pid
|
||||
}
|
||||
gpuCheckWait = 0
|
||||
t.Cleanup(func() { gpuProcesses, processGroup, gpuCheckWait = oldProcs, oldGroup, oldWait })
|
||||
|
||||
a := &Agent{log: slog.Default()}
|
||||
spec := engines.ShardSpec{ShardID: "s", GPUIndices: []int{0, 1}}
|
||||
eng := fallbackEngine{}
|
||||
|
||||
// On no GPU at all: refused, saying what the engine sees.
|
||||
err := a.onItsGPUs(context.Background(), eng, spec, 500)
|
||||
if err == nil || !strings.Contains(err.Error(), "computing on the CPU") || !strings.Contains(err.Error(), "lists no GPU") {
|
||||
t.Fatalf("a shard on no GPU was let through: %v", err)
|
||||
}
|
||||
// On one of its two: refused, naming the one it is missing.
|
||||
procs[0] = []int{500}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil || !strings.Contains(err.Error(), "GPU 1 ") {
|
||||
t.Fatalf("a shard on half its GPUs: %v", err)
|
||||
}
|
||||
// Another process on GPU 1 is not the shard.
|
||||
procs[1] = []int{900}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil {
|
||||
t.Fatal("another process's memory was taken for the shard's")
|
||||
}
|
||||
// A child of the shard holding GPU 1 is the shard.
|
||||
procs[1] = []int{900, 512}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err != nil {
|
||||
t.Fatalf("a shard on both its GPUs: %v", err)
|
||||
}
|
||||
|
||||
// An engine that fails rather than falling back is not checked, nor is a
|
||||
// shard with no GPUs.
|
||||
procs = map[int][]int{}
|
||||
if err := a.onItsGPUs(context.Background(), struct{ engines.Engine }{}, spec, 500); err != nil {
|
||||
t.Errorf("checked an engine that does not fall back: %v", err)
|
||||
}
|
||||
if err := a.onItsGPUs(context.Background(), eng, engines.ShardSpec{}, 500); err != nil {
|
||||
t.Errorf("checked a CPU shard: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCheckWaitsForTheShardToShowUpOnItsGPU(t *testing.T) {
|
||||
calls := 0
|
||||
oldProcs, oldWait := gpuProcesses, gpuCheckWait
|
||||
gpuProcesses = func(context.Context, string) (map[int][]int, error) {
|
||||
calls++
|
||||
if calls < 2 {
|
||||
return map[int][]int{}, nil
|
||||
}
|
||||
return map[int][]int{0: {77}}, nil
|
||||
}
|
||||
gpuCheckWait = 5 * time.Second
|
||||
t.Cleanup(func() { gpuProcesses, gpuCheckWait = oldProcs, oldWait })
|
||||
|
||||
a := &Agent{log: slog.Default()}
|
||||
if err := a.onItsGPUs(context.Background(), fallbackEngine{}, engines.ShardSpec{GPUIndices: []int{0}}, 77); err != nil {
|
||||
t.Fatalf("nvidia-smi lagging by a second failed the shard: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -46,9 +46,30 @@ func (a *Agent) engineReportLocked() map[string]string {
|
||||
out = map[string]string{}
|
||||
}
|
||||
maps.Copy(out, a.installedVers)
|
||||
maps.DeleteFunc(out, func(e, _ string) bool { return a.disabled(e) })
|
||||
return out
|
||||
}
|
||||
|
||||
// byHand reports whether agent.yaml names this engine's binary, which opts
|
||||
// it out of being installed.
|
||||
func (a *Agent) byHand(engine string) bool {
|
||||
if a.reg == nil {
|
||||
return false
|
||||
}
|
||||
e, err := a.reg.Get(engine)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
c, ok := e.(engines.Configured)
|
||||
return ok && c.Configured()
|
||||
}
|
||||
|
||||
// disabled reports whether agent.yaml says this node never runs an engine:
|
||||
// not installed when asked, and not advertised when found.
|
||||
func (a *Agent) disabled(engine string) bool {
|
||||
return slices.Contains(a.cfg.Engines.Disabled, engine)
|
||||
}
|
||||
|
||||
// provisionEvery is how often installed engines are checked again. A library
|
||||
// removed by an unrelated upgrade is found here rather than by the next
|
||||
// replica to start.
|
||||
@@ -58,7 +79,16 @@ const provisionEvery = 10 * time.Minute
|
||||
func (a *Agent) setProvision(p *llmgwv1.Provision) {
|
||||
wants := []provision.Want{}
|
||||
for _, e := range p.GetEngines() {
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
|
||||
if a.disabled(e.GetEngine()) {
|
||||
continue // a control plane older than the node's say in it
|
||||
}
|
||||
if a.byHand(e.GetEngine()) {
|
||||
// The operator named a binary for it: a local build, a CUDA
|
||||
// runtime of their own. Installing one beside it would win —
|
||||
// the installed build is used first — and quietly replace theirs.
|
||||
continue
|
||||
}
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion(), Variant: e.GetVariant()})
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.provWants = wants
|
||||
|
||||
@@ -27,6 +27,9 @@ import (
|
||||
type Want struct {
|
||||
Engine string
|
||||
Version string
|
||||
// Variant is a build chosen for this node ("cu129"); empty leaves it to
|
||||
// the node's hardware.
|
||||
Variant string
|
||||
}
|
||||
|
||||
// The states an engine's installation can be in.
|
||||
@@ -135,7 +138,12 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
"; install it by hand and name it in the agent's configuration"
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, in.Host)
|
||||
if w.Variant != "" && !validVersion.MatchString(w.Variant) {
|
||||
st.State = Unsupported
|
||||
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
|
||||
return st
|
||||
}
|
||||
builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Unsupported, err.Error()
|
||||
var nh NotHere
|
||||
@@ -144,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
}
|
||||
return st
|
||||
}
|
||||
st.Variant = build.Variant
|
||||
dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
|
||||
|
||||
// Installed before: check it still runs, and trust nothing else about it.
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
|
||||
// Installed before: whichever candidate is, best first, checked that it
|
||||
// still runs and trusted for nothing else.
|
||||
for _, build := range builds {
|
||||
dir := in.dirFor(w, build)
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
|
||||
continue
|
||||
}
|
||||
st := st.as(w, build)
|
||||
// The packages are checked too. Removing one is exactly the kind of
|
||||
// change an unrelated upgrade makes, and ldd below would catch it only
|
||||
// as a missing library with no hint of which package held it.
|
||||
// change an unrelated upgrade makes, and ldd below would catch it
|
||||
// only as a missing library with no hint of which package held it.
|
||||
if err := in.packages(ctx, build.Packages, &st); err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
@@ -161,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
in.log().Warn("an installed engine no longer runs; installing it again",
|
||||
"engine", w.Engine, "version", w.Version, "err", err)
|
||||
"engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
|
||||
break
|
||||
}
|
||||
|
||||
if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
|
||||
st.State, st.Detail, st.Binary = Failed, err.Error(), ""
|
||||
return st
|
||||
// Each candidate in turn until one installs and is seen to work: a build
|
||||
// the table was wrong about — a driver too old for it after all, a GPU
|
||||
// it has no kernels for — costs its download, and the next is tried.
|
||||
var tried []string
|
||||
for _, build := range builds {
|
||||
st := st.as(w, build)
|
||||
if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
in.log().Warn("an engine build did not work here; trying the next",
|
||||
"engine", w.Engine, "variant", build.Variant, "err", err)
|
||||
continue
|
||||
}
|
||||
bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
|
||||
if err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
bin, version, err := in.verify(ctx, dir, build)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
st.State, st.Binary = Failed, ""
|
||||
st.Detail = strings.Join(tried, "; then ")
|
||||
if len(builds) == 1 {
|
||||
_, st.Detail, _ = strings.Cut(st.Detail, ": ")
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
return st
|
||||
}
|
||||
|
||||
// dirFor is where a build of an engine is installed: by the version it is
|
||||
// and its variant, so two builds of one version sit side by side.
|
||||
func (in *Installer) dirFor(w Want, b Build) string {
|
||||
return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
|
||||
}
|
||||
|
||||
// versionOr is the version a build is: its own, when it is not the one asked
|
||||
// for.
|
||||
func (b Build) versionOr(asked string) string {
|
||||
if b.Version != "" {
|
||||
return b.Version
|
||||
}
|
||||
return asked
|
||||
}
|
||||
|
||||
// as is a status for one build of what was asked for.
|
||||
func (st Status) as(w Want, b Build) Status {
|
||||
st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
|
||||
return st
|
||||
}
|
||||
|
||||
func (in *Installer) ready(st Status, b Build, bin, version string) Status {
|
||||
st.State, st.Binary = Ready, bin
|
||||
st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
|
||||
if b.Note != "" {
|
||||
st.Detail += "; " + b.Note
|
||||
}
|
||||
var missing []string
|
||||
for _, f := range b.CPUFlags {
|
||||
if !in.Host.HasCPUFlag(f) {
|
||||
@@ -299,9 +351,55 @@ func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, vers
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
|
||||
}
|
||||
if b.GPUBackend != "" {
|
||||
if err := in.verifyGPUBackend(tctx, dir, bin, b); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
}
|
||||
return bin, firstVersionLine(out), nil
|
||||
}
|
||||
|
||||
// verifyGPUBackend checks a GPU build can compute on this node's GPUs.
|
||||
//
|
||||
// The binary's own ldd says nothing about it: the GPU backend is a library
|
||||
// the server opens at run time, and when it cannot — cuBLAS missing, a driver
|
||||
// too old — the server carries on with its CPU backend. So the build is asked
|
||||
// which devices it sees, and when it sees none, ldd on the backend says why.
|
||||
func (in *Installer) verifyGPUBackend(ctx context.Context, dir, bin string, b Build) error {
|
||||
out, err := in.run(ctx, nil, bin, "--list-devices")
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s --list-devices failed: %v: %s", b.Binary, err, lastLine(out))
|
||||
}
|
||||
if listsDevice(string(out)) {
|
||||
return nil
|
||||
}
|
||||
why := "its GPU backend did not load"
|
||||
if lib, err := findFile(dir, b.GPUBackend); err != nil {
|
||||
why = "the build has no " + b.GPUBackend
|
||||
} else if out, err := in.run(ctx, nil, "ldd", lib); err == nil {
|
||||
if missing := lddMissing(string(out)); len(missing) > 0 {
|
||||
why = b.GPUBackend + " cannot load: " + strings.Join(missing, "; ")
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("the %s build of %s sees no GPU, and would run on the CPU: %s", b.Variant, b.Binary, why)
|
||||
}
|
||||
|
||||
// listsDevice reads llama-server --list-devices: a device line after
|
||||
// "Available devices:", which holds "(none)" when there is none.
|
||||
func listsDevice(out string) bool {
|
||||
listing := false
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
switch {
|
||||
case strings.HasPrefix(line, "Available devices"):
|
||||
listing = true
|
||||
case listing && line != "" && line != "(none)":
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// packages installs the operating system packages a build needs.
|
||||
//
|
||||
// Only those missing, and from the node's own package sources: a package the
|
||||
|
||||
@@ -89,7 +89,11 @@ type fakeSystem struct {
|
||||
installed map[string]bool
|
||||
tools map[string]bool
|
||||
lddOut string
|
||||
calls []string
|
||||
// devices is what llama-server --list-devices prints, and backendLdd
|
||||
// what ldd says of its GPU backend.
|
||||
devices string
|
||||
backendLdd string
|
||||
calls []string
|
||||
// locked is how many more package commands find the lock held.
|
||||
locked int
|
||||
}
|
||||
@@ -127,8 +131,12 @@ func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...str
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
case name == "ldd" && strings.HasSuffix(args[0], "libggml-cuda.so"):
|
||||
return []byte(f.backendLdd), nil
|
||||
case name == "ldd":
|
||||
return []byte(f.lddOut), nil
|
||||
case strings.HasSuffix(name, "llama-server") && args[0] == "--list-devices":
|
||||
return []byte(f.devices), nil
|
||||
case strings.HasSuffix(name, "llama-server"):
|
||||
return []byte("version: 10991 (abc)\n"), nil
|
||||
}
|
||||
@@ -247,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
|
||||
func TestTheBuildFollowsTheHardware(t *testing.T) {
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
b, err := chooseLlamaCpp("b1", gpu)
|
||||
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
|
||||
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
|
||||
got, err := llamaCppCandidates("b1", "", gpu)
|
||||
if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
|
||||
t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
|
||||
}
|
||||
if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
|
||||
}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
host discovery.Host
|
||||
want []string
|
||||
}{
|
||||
// Driver 580 and Turing: CUDA 13.3 first.
|
||||
{"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
|
||||
// A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
|
||||
{"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
|
||||
// An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
|
||||
{"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
|
||||
// A Pascal card: CUDA 12.8, not 13.3.
|
||||
{"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
|
||||
// arm64 with a GPU: never the CPU build.
|
||||
{"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
|
||||
} {
|
||||
got, err := llamaCppCandidates("b1", "", c.host)
|
||||
var names []string
|
||||
for _, b := range got {
|
||||
names = append(names, b.Variant)
|
||||
}
|
||||
if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
|
||||
t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
|
||||
}
|
||||
}
|
||||
|
||||
slow := ubuntu
|
||||
slow.CPUFlags = nil
|
||||
@@ -388,3 +421,32 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
|
||||
t.Errorf("link %q %v", target, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
|
||||
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
|
||||
// loader looks, and llama-server serving a GPU placement on the CPU.
|
||||
cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
|
||||
rel := newRelease(map[string][]byte{
|
||||
"llama-b1-" + cuda: tarGz(t, map[string]string{
|
||||
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
|
||||
"cudart-llama-b1-" + cuda: tarGz(t, map[string]string{"libcudart.so.12": "x"}, nil),
|
||||
})
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
|
||||
devices: "Available devices:\n (none)\n",
|
||||
backendLdd: "\tlibcublas.so.12 => not found\n\tlibcudart.so.12 => /x/libcudart.so.12\n"}
|
||||
in, _ := installer(t, rel.serve(t), sys, gpu)
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "sees no GPU") ||
|
||||
!strings.Contains(got[0].Detail, "libcublas.so.12 is not installed") {
|
||||
t.Fatalf("expected a failure naming the missing library: %+v", got[0])
|
||||
}
|
||||
|
||||
// The same build seeing its GPU is installed.
|
||||
sys.devices = "Available devices:\n CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}}); got[0].State != Ready {
|
||||
t.Fatalf("a CUDA build that sees its GPU: %+v", got[0])
|
||||
}
|
||||
}
|
||||
@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
|
||||
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
|
||||
defer cancel()
|
||||
// The version this build is, which for a GPU older than every build of
|
||||
// the one asked for is an older one: its publication date is the one
|
||||
// the dependencies are resolved as of.
|
||||
w.Version = b.versionOr(w.Version)
|
||||
|
||||
if err := in.packages(ctx, b.Packages, st); err != nil {
|
||||
return err
|
||||
@@ -86,28 +90,67 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
python := filepath.Join(tmp, "bin", "python")
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
|
||||
reqs := b.Requirements
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
// refuses a download that does not match it.
|
||||
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var asset, sum string
|
||||
for name, s := range sums {
|
||||
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
|
||||
asset, sum = name, s
|
||||
}
|
||||
}
|
||||
if asset == "" {
|
||||
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
|
||||
b.Wheel.Tag + "/" + asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
// is taken from whichever of the two has the version resolved — both are
|
||||
// the publishers' own — and each file is still pinned by its hash.
|
||||
var index []string
|
||||
if b.ExtraIndex != "" {
|
||||
index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
|
||||
}
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
|
||||
lock := filepath.Join(tmp, lockName)
|
||||
args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock, "-"}
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
|
||||
args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock}, index...)
|
||||
args = append(args, "-")
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
|
||||
strings.Join(b.Requirements, " "))
|
||||
if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
|
||||
strings.Join(reqs, " "))
|
||||
install := append([]string{"pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock}, index...)
|
||||
if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
in.progress(st, "checking that %s runs", w.Engine)
|
||||
if _, _, err := in.verify(ctx, tmp, b); err != nil {
|
||||
return err
|
||||
}
|
||||
if b.GPUCheck && in.Host.NVIDIADriver != "" {
|
||||
in.progress(st, "checking that %s (%s) computes on this node's GPU", w.Engine, b.Variant)
|
||||
if err := in.gpuCheck(ctx, python, w, b); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
m, _ := json.MarshalIndent(markerFile{
|
||||
Engine: w.Engine, Version: w.Version, Variant: b.Variant,
|
||||
Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
|
||||
Binary: filepath.Join("bin", b.Binary), Requirements: reqs,
|
||||
ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
|
||||
}, "", " ")
|
||||
if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
|
||||
@@ -203,3 +246,26 @@ func tail(b []byte) string {
|
||||
}
|
||||
return strings.Join(lines, " / ")
|
||||
}
|
||||
|
||||
// gpuProbe is what the environment is asked to run on the GPU: one small
|
||||
// operation, which needs the torch to have kernels for the card.
|
||||
const gpuProbe = `import torch
|
||||
assert torch.cuda.is_available(), "torch sees no CUDA device"
|
||||
x = torch.ones(64, device="cuda")
|
||||
print(float((x * 2).sum()), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))`
|
||||
|
||||
// gpuCheck runs gpuProbe in a freshly installed environment.
|
||||
func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Build) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
|
||||
defer cancel()
|
||||
out, err := in.run(ctx, nil, python, "-c", gpuProbe)
|
||||
if err != nil {
|
||||
hint := ""
|
||||
if strings.Contains(string(out), "no kernel image") {
|
||||
hint = "; this build has no kernels for this GPU, so try the other build for this node"
|
||||
}
|
||||
return fmt.Errorf("%s %s (%s) installed but cannot compute on this node's GPU: %s%s",
|
||||
w.Engine, w.Version, b.Variant, tail(out), hint)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package provision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
@@ -23,6 +24,10 @@ type fakePython struct {
|
||||
pmu sync.Mutex
|
||||
commands []string
|
||||
failPip bool
|
||||
// noKernel makes the GPU check fail as a torch without kernels for the
|
||||
// card does: in every build, or in noKernelIn only ("-cu13.").
|
||||
noKernel bool
|
||||
noKernelIn string
|
||||
}
|
||||
|
||||
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
@@ -43,6 +48,11 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
|
||||
if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
|
||||
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
|
||||
}
|
||||
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
|
||||
}
|
||||
return f.fakeSystem.run(ctx, env, name, args...)
|
||||
}
|
||||
@@ -59,6 +69,8 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
uv := newRelease(map[string][]byte{
|
||||
"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
|
||||
map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
|
||||
// The fake GitHub serves every repository's release from this one.
|
||||
"vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl": []byte("the cu129 wheel"),
|
||||
})
|
||||
gh := uv.serve(t)
|
||||
mux := http.NewServeMux()
|
||||
@@ -66,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
|
||||
{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
|
||||
})
|
||||
mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
|
||||
})
|
||||
pypi := httptest.NewServer(mux)
|
||||
t.Cleanup(pypi.Close)
|
||||
return &Installer{
|
||||
@@ -130,8 +145,10 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
says string
|
||||
}{
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
|
||||
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
// No vLLM was ever built for Pascal.
|
||||
{"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -182,3 +199,114 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
|
||||
t.Errorf("got %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
|
||||
// 12.9 build, from the project's GitHub release, with the torch PyTorch
|
||||
// publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, gh := pythonInstaller(t, volta, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("a Turing node on driver 550: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
|
||||
t.Errorf("not installed under its own build's name: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
wheel := "vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
|
||||
for _, want := range []string{
|
||||
// Pinned to the digest GitHub publishes for the wheel.
|
||||
"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
|
||||
strings.TrimPrefix(gh.digests[wheel], "sha256:"),
|
||||
"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
|
||||
"--require-hashes --no-deps -r",
|
||||
// Computing on the card, not only importing.
|
||||
"/bin/python -c import torch",
|
||||
} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("no %q in:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
|
||||
t.Errorf("installed without the index its torch comes from: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
|
||||
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
|
||||
}
|
||||
if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
|
||||
t.Error("a build that does not exist was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernel: true}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "no kernels for this GPU") {
|
||||
t.Fatalf("a build without kernels for the card: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu13", marker)); err == nil {
|
||||
t.Error("marked installed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
// A machine vLLM is not published for is refused, not given the other
|
||||
// build to fail with.
|
||||
if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
!strings.Contains(err.Error(), "no Linux build") {
|
||||
t.Errorf("an unknown architecture: %v", err)
|
||||
}
|
||||
if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
// Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
|
||||
got, err := vllmCandidates("0.30.0", "", gpuNode)
|
||||
if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
|
||||
t.Errorf("candidates on a current node: %+v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
|
||||
// 0.30.0 asked for; no build of it has Volta kernels. The node installs
|
||||
// 0.19.1, the newest that has, and says so.
|
||||
v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, v100, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
|
||||
!strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
|
||||
t.Fatalf("a V100: %+v", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
|
||||
t.Errorf("not installed as what it is: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
|
||||
t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
|
||||
}
|
||||
// Chosen cu13, which 0.19.1 has none of: refused, saying why.
|
||||
if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
|
||||
t.Error("a CUDA 13 build was offered to a V100")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
|
||||
// The table says CUDA 13 runs here; the card says otherwise. The next
|
||||
// build is installed rather than the node going without.
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Variant != "cu129" {
|
||||
t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -43,12 +43,41 @@ type Build struct {
|
||||
// VersionArgs is what Binary is run with to prove the build works, when
|
||||
// --version is not it.
|
||||
VersionArgs []string
|
||||
// GPUBackend, when set, is the shared library the build computes on GPUs
|
||||
// with, loaded at run time rather than linked: "libggml-cuda.so". The
|
||||
// build must list a device with it, or it is not installed — a llama.cpp
|
||||
// whose CUDA backend cannot load runs on the CPU without complaint.
|
||||
GPUBackend string
|
||||
// Wheel, when set, is the engine itself as a wheel from a GitHub
|
||||
// release rather than from PyPI, checked against the digest GitHub
|
||||
// publishes for it. ExtraIndex is a package index besides PyPI that the
|
||||
// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
|
||||
Wheel *Wheel
|
||||
ExtraIndex string
|
||||
// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
|
||||
// compute something on one before the build counts as installed. A build
|
||||
// with no kernels for the card — a GPU generation its torch dropped —
|
||||
// imports and prints its version, and fails on its first model.
|
||||
GPUCheck bool
|
||||
// Version is the engine version this build is, when it is not the one
|
||||
// asked for: the newest built for this machine's GPUs, where the one
|
||||
// asked for has no kernels for them. Empty is the version asked for.
|
||||
Version string
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
Project, Repo, Tag, Prefix, Suffix string
|
||||
}
|
||||
|
||||
// Recipe knows how to install one engine.
|
||||
type Recipe struct {
|
||||
// Repo is the GitHub repository whose releases carry the builds; empty
|
||||
@@ -56,8 +85,12 @@ type Recipe struct {
|
||||
Repo string
|
||||
// PyPI is the project an engine installed from PyPI is published as.
|
||||
PyPI string
|
||||
// Choose picks the build for a machine, or says why none fits.
|
||||
Choose func(version string, h discovery.Host) (Build, error)
|
||||
// Candidates are the builds that may run on a machine, best first, or
|
||||
// why none can. The installer takes the first that installs and is seen
|
||||
// to work — computing on the GPU, where there is one — so a build this
|
||||
// table is wrong about costs a download, not a node. variant is a build
|
||||
// chosen for the node; empty leaves the choice to the machine.
|
||||
Candidates func(version, variant string, h discovery.Host) ([]Build, error)
|
||||
}
|
||||
|
||||
// Recipes are the engines this agent can install.
|
||||
@@ -66,10 +99,10 @@ type Recipe struct {
|
||||
// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
|
||||
// checked against its published checksum.
|
||||
var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
|
||||
"vllm": {PyPI: "vllm", Choose: chooseVLLM},
|
||||
"sglang": {PyPI: "sglang", Choose: chooseSGLang},
|
||||
"uv": {Repo: "astral-sh/uv", Choose: chooseUV},
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
// Installable are the engines this agent can install, as it tells the
|
||||
@@ -99,7 +132,15 @@ type NotHere struct{ Reason string }
|
||||
|
||||
func (e NotHere) Error() string { return e.Reason }
|
||||
|
||||
func chooseUV(version string, h discovery.Host) (Build, error) {
|
||||
func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
|
||||
b, err := chooseUV(version, h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []Build{b}, nil
|
||||
}
|
||||
|
||||
func chooseUV(_ string, h discovery.Host) (Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -116,33 +157,227 @@ func chooseUV(version string, h discovery.Host) (Build, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
|
||||
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
|
||||
const minDriverForCUDA13 = 580
|
||||
// What builds need of the NVIDIA side of a machine, read from what each
|
||||
// project publishes and how it was built:
|
||||
//
|
||||
// - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
|
||||
// by CUDA's minor version compatibility — except code shipped only as
|
||||
// PTX, which the driver compiles and so needs a driver as new as the
|
||||
// CUDA it was compiled with.
|
||||
// - Which GPUs a build has kernels for is the build's own list, not what
|
||||
// its CUDA supports.
|
||||
const (
|
||||
minDriverForCUDA13 = 580
|
||||
minDriverForCUDA12 = 525
|
||||
)
|
||||
|
||||
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
// gpuNeeds is what one build needs of a machine's GPUs and driver.
|
||||
type gpuNeeds struct {
|
||||
driver int
|
||||
cc float64
|
||||
}
|
||||
|
||||
// cudaGate says whether a node can run an engine built on CUDA 13.
|
||||
func cudaGate(engine string, h discovery.Host) error {
|
||||
// fits says why a machine cannot run a build, or nil.
|
||||
func (n gpuNeeds) fits(what string, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
|
||||
return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
|
||||
return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// candidate is a build and what it needs.
|
||||
type candidate struct {
|
||||
b Build
|
||||
needs gpuNeeds
|
||||
what string
|
||||
}
|
||||
|
||||
// pick keeps the candidates a machine can run, in order, and those a
|
||||
// variant chosen for the node names; with none left, it says why each went.
|
||||
func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
|
||||
var out []Build
|
||||
var why []string
|
||||
for _, c := range cands {
|
||||
if variant != "" && c.b.Variant != variant {
|
||||
continue
|
||||
}
|
||||
if err := c.needs.fits(c.what, h); err != nil {
|
||||
why = append(why, err.Error())
|
||||
continue
|
||||
}
|
||||
out = append(out, c.b)
|
||||
}
|
||||
switch {
|
||||
case len(out) > 0:
|
||||
return out, nil
|
||||
case len(why) == 0 && variant != "":
|
||||
var names []string
|
||||
for _, c := range cands {
|
||||
names = append(names, c.b.Variant)
|
||||
}
|
||||
return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
|
||||
}
|
||||
return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
|
||||
}
|
||||
|
||||
// --- vLLM ---
|
||||
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for drivers CUDA 13 does not run on.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
|
||||
// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
|
||||
// "+cu129" beside a CUDA 13 one.
|
||||
const vllmCUDA13Since = "0.20"
|
||||
|
||||
// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
|
||||
// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
|
||||
// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
|
||||
// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
|
||||
// 0.20 and 8.0 since.
|
||||
func vllmMinCompute(version, arch string) float64 {
|
||||
old := versionBefore(version, vllmCUDA13Since)
|
||||
switch {
|
||||
case arch == "arm64" && old:
|
||||
return 8.7
|
||||
case arch == "arm64":
|
||||
return 8.0
|
||||
case old:
|
||||
return 7.0
|
||||
}
|
||||
return 7.5
|
||||
}
|
||||
|
||||
// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
|
||||
// on an architecture, for GPUs older than every build of the version asked
|
||||
// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
|
||||
// for a GPU no vLLM was built for.
|
||||
var vllmNewestFor = []struct {
|
||||
arch string
|
||||
from float64 // the oldest GPU it covers
|
||||
version string
|
||||
}{
|
||||
{"amd64", 7.0, "0.19.1"},
|
||||
}
|
||||
|
||||
// vllmCandidates are the vLLM builds for a machine.
|
||||
//
|
||||
// The version asked for, in the build the machine runs: CUDA 13 where the
|
||||
// driver allows, the same version's CUDA 12.9 build where it does not. One
|
||||
// version across the cluster wherever the GPUs allow it, because the flags
|
||||
// llmgw builds are checked against it. Where they do not — GPUs older than
|
||||
// every build of that version — the newest version built for them, said so
|
||||
// beside the installation: vLLM on a V100 at all, rather than not.
|
||||
func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
arch, err := gpuArch("vLLM", h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v, note := version, ""
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
|
||||
older := ""
|
||||
for _, n := range vllmNewestFor {
|
||||
if n.arch == h.Arch && cc >= n.from {
|
||||
older = n.version
|
||||
}
|
||||
}
|
||||
if older == "" {
|
||||
return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
|
||||
"the node serves llama.cpp models", cc, arch)
|
||||
}
|
||||
v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
|
||||
}
|
||||
pypi := func(variant string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + v},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
Note: note,
|
||||
}
|
||||
if v != version {
|
||||
b.Version = v
|
||||
}
|
||||
return b
|
||||
}
|
||||
minCC := vllmMinCompute(v, h.Arch)
|
||||
if versionBefore(v, vllmCUDA13Since) {
|
||||
// PyPI's own build is CUDA 12, and is the one there is; asked for
|
||||
// cu129, this is it.
|
||||
if variant == "cu129" {
|
||||
variant = "cu12"
|
||||
}
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
|
||||
}, h)
|
||||
}
|
||||
cu129 := pypi("cu129")
|
||||
cu129.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
cu129.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
|
||||
Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
|
||||
}
|
||||
cu129.ExtraIndex = pytorchCU129
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
|
||||
}, h)
|
||||
}
|
||||
|
||||
// gpuArch says whether a Python GPU engine is published for this machine
|
||||
// and whether it has an NVIDIA GPU at all.
|
||||
func gpuArch(engine string, h discovery.Host) (string, error) {
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
|
||||
return "", fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
|
||||
}
|
||||
if h.NVIDIADriver == "" {
|
||||
return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
|
||||
return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, minDriverForCUDA13, h.NVIDIADriver)
|
||||
return h.Arch, nil
|
||||
}
|
||||
|
||||
// --- SGLang ---
|
||||
|
||||
// sglangCUDA13Since is the first SGLang built for CUDA 13.
|
||||
const sglangCUDA13Since = "0.5"
|
||||
|
||||
func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("SGLang", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
|
||||
return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
|
||||
engine, minComputeForCUDA13, cc)
|
||||
b := Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
return nil
|
||||
needs := gpuNeeds{minDriverForCUDA13, 7.5}
|
||||
if versionBefore(version, sglangCUDA13Since) {
|
||||
b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
|
||||
}
|
||||
return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
@@ -150,35 +385,27 @@ func cudaGate(engine string, h discovery.Host) error {
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
func chooseVLLM(version string, h discovery.Host) (Build, error) {
|
||||
if err := cudaGate("vLLM", h); err != nil {
|
||||
return Build{}, err
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
func versionBefore(a, b string) bool {
|
||||
if b == "" {
|
||||
return false
|
||||
}
|
||||
return Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func chooseSGLang(version string, h discovery.Host) (Build, error) {
|
||||
if err := cudaGate("SGLang", h); err != nil {
|
||||
return Build{}, err
|
||||
pa, pb := strings.Split(strings.TrimPrefix(a, "v"), "."), strings.Split(b, ".")
|
||||
for i := 0; i < max(len(pa), len(pb)); i++ {
|
||||
x, y := 0, 0
|
||||
if i < len(pa) {
|
||||
x = leadingInt(pa[i])
|
||||
}
|
||||
if i < len(pb) {
|
||||
y = leadingInt(pb[i])
|
||||
}
|
||||
if x != y {
|
||||
return x < y
|
||||
}
|
||||
}
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
return Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
return false
|
||||
}
|
||||
|
||||
// leadingInt is the number a version string starts with: 580 of "580.65.06".
|
||||
@@ -193,17 +420,30 @@ func leadingInt(s string) int {
|
||||
return n
|
||||
}
|
||||
|
||||
// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
|
||||
// carries a matching cudart archive, so the node needs a driver and nothing
|
||||
// else — no toolkit.
|
||||
const cudaForLlamaCpp = "12.8"
|
||||
// --- llama.cpp ---
|
||||
|
||||
func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them
|
||||
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
|
||||
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
|
||||
// name decides nothing here. The build is tried everywhere, and ldd says
|
||||
// whether this system can load it — in the words of the missing symbol.
|
||||
// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
|
||||
// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
|
||||
// carries as PTX, which a driver older than the CUDA it came from cannot
|
||||
// compile.
|
||||
var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
|
||||
|
||||
// llamaCppCandidates are the llama.cpp builds for a machine.
|
||||
//
|
||||
// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
|
||||
// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
|
||||
// the GPUs it ships as PTX, 525 for those it ships as machine code), and
|
||||
// Vulkan, which runs on any card whose driver has Vulkan — the generations
|
||||
// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
|
||||
// a GPU placement served on the CPU, which is what installing checks for.
|
||||
// The CPU build where there is no NVIDIA GPU.
|
||||
//
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them is
|
||||
// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
|
||||
// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
|
||||
// nothing here. The build is tried everywhere, and ldd says whether this
|
||||
// system can load it — in the words of the missing symbol.
|
||||
func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -211,35 +451,53 @@ func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
|
||||
case "arm64":
|
||||
arch = "arm64"
|
||||
default:
|
||||
return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
|
||||
b := Build{
|
||||
Variant: "cpu",
|
||||
Assets: []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
|
||||
// Everything else the build links is in the archive or in any current
|
||||
// glibc system; OpenMP's runtime is the one a cloud image lacks, and
|
||||
// the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the best
|
||||
// one, down to SSE4.2. Below AVX2 it runs, several times slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
// The CUDA build is published for x64 at this version; on arm64 it is a
|
||||
// newer CUDA that the driver may not support, so an arm64 GPU node runs
|
||||
// the CPU build until someone decides otherwise.
|
||||
if h.NVIDIADriver != "" && arch == "x64" {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
|
||||
b.Variant = "cuda-" + cudaForLlamaCpp
|
||||
b.Assets = []string{
|
||||
fmt.Sprintf("llama-%s-%s", version, name),
|
||||
fmt.Sprintf("cudart-llama-%s-%s", version, name),
|
||||
base := func(variant string, assets ...string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Assets: assets,
|
||||
// Everything else the build links is in the archive or in any
|
||||
// current glibc system; OpenMP's runtime is the one a cloud image
|
||||
// lacks, and the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the
|
||||
// best one, down to SSE4.2. Below AVX2 it runs, several times
|
||||
// slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
return b
|
||||
}
|
||||
return b, nil
|
||||
if h.NVIDIADriver == "" {
|
||||
return pick("llama.cpp", variant, []candidate{
|
||||
{base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
|
||||
}, h)
|
||||
}
|
||||
cuda := func(cv string) Build {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
|
||||
// The release carries the CUDA runtime beside the build, so the node
|
||||
// needs a driver and nothing else — no toolkit.
|
||||
b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
return b
|
||||
}
|
||||
vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
|
||||
vulkan.GPUBackend = "libggml-vulkan.so"
|
||||
vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
|
||||
|
||||
cuda12Driver := 570
|
||||
if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
|
||||
cuda12Driver = minDriverForCUDA12
|
||||
}
|
||||
cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
|
||||
if arch == "x64" {
|
||||
cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
|
||||
}
|
||||
cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
|
||||
return pick("llama.cpp", variant, cands, h)
|
||||
}
|
||||
|
||||
func orUnknown(s string) string {
|
||||
|
||||
@@ -2,6 +2,8 @@ package weights
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -26,8 +28,9 @@ type manifest struct {
|
||||
Model string `json:"model"`
|
||||
Version string `json:"version"`
|
||||
Files []struct {
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
} `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
@@ -87,6 +90,25 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error)
|
||||
}
|
||||
}
|
||||
|
||||
// Each file against the checksum the mirror took when it fetched it,
|
||||
// before the set is called complete: a shard damaged on the way loads as
|
||||
// a model that answers nonsense, not as an error.
|
||||
for _, file := range m.Files {
|
||||
if file.SHA256 == "" {
|
||||
continue
|
||||
}
|
||||
target := filepath.Join(dir, file.Name)
|
||||
got, err := fileSHA256(target)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !strings.EqualFold(got, file.SHA256) {
|
||||
_ = os.Remove(target)
|
||||
return "", fmt.Errorf("%s%s: sha256 %s, the mirror has %s; removed, to be fetched again",
|
||||
source, file.Name, got, file.SHA256)
|
||||
}
|
||||
}
|
||||
|
||||
if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -108,6 +130,12 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
|
||||
return manifest{}, fmt.Errorf("asking %s what it holds: %w", source, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode == http.StatusConflict {
|
||||
// Still downloading on the mirror, or a shard missing: said in the
|
||||
// mirror's words, which name what is not there yet.
|
||||
why, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
|
||||
return manifest{}, fmt.Errorf("%s is not complete on the mirror yet: %s", source, strings.TrimSpace(string(why)))
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return manifest{}, fmt.Errorf("%s answered %d", source, resp.StatusCode)
|
||||
}
|
||||
@@ -127,3 +155,16 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func fileSHA256(path string) (string, error) {
|
||||
f, err := os.Open(path) //nolint:gosec // a file this fetcher wrote
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
if _, err := io.Copy(h, f); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), nil
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
package weights
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -165,3 +167,40 @@ func TestOnlyATrailingSlashMeansASet(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetFileThatDoesNotMatchTheMirrorsChecksumIsRefused(t *testing.T) {
|
||||
good := "the shard as mirrored"
|
||||
sum := sha256.Sum256([]byte(good))
|
||||
served := "the shard damaged on the way"
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/") {
|
||||
fmt.Fprintf(w, `{"model":"m","version":"v","files":[{"name":"s.safetensors","bytes":%d,"sha256":%q}]}`,
|
||||
len(served), hex.EncodeToString(sum[:]))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(served))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
f := &Fetcher{CacheDir: t.TempDir()}
|
||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err == nil || !strings.Contains(err.Error(), "sha256") {
|
||||
t.Fatalf("a damaged shard was accepted: %v", err)
|
||||
}
|
||||
served = good
|
||||
// The size differs, so it is fetched again; this time it matches.
|
||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
|
||||
t.Fatalf("the shard as mirrored: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetStillDownloadingOnTheMirrorSaysSo(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.Error(w, "m/v: this version is not complete: still downloading model-00002-of-00002.safetensors", http.StatusConflict)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
f := &Fetcher{CacheDir: t.TempDir()}
|
||||
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
|
||||
if err == nil || !strings.Contains(err.Error(), "not complete on the mirror yet") ||
|
||||
!strings.Contains(err.Error(), "model-00002-of-00002") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
}
|
||||
@@ -143,6 +143,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
newEngineCheckCmd(),
|
||||
newEngineBuildCmd(client),
|
||||
newToolCheckCmd(),
|
||||
newWeightsCmd(client, &asJSON),
|
||||
newCapacityCmd(client, &asJSON),
|
||||
|
||||
@@ -12,8 +12,9 @@ import (
|
||||
|
||||
func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
var (
|
||||
models []string
|
||||
all bool
|
||||
models []string
|
||||
all bool
|
||||
inPlace bool
|
||||
)
|
||||
c := &cobra.Command{
|
||||
Use: "restart [REPLICA...]",
|
||||
@@ -22,10 +23,14 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
"an edited profile, a new engine version.\n\n" +
|
||||
"Each replacement is placed beside the replica it replaces, and the old one\n" +
|
||||
"keeps serving until the new one is ready. Replicas of one model are\n" +
|
||||
"replaced one at a time. Where nothing else has room for a second copy —\n" +
|
||||
"the usual case on a small CPU fleet — the old one is stopped after the\n" +
|
||||
"drain grace period (5 minutes by default) and the new one starts in its\n" +
|
||||
"place; a model with only that replica is unserved while it loads.\n\n" +
|
||||
"replaced one at a time.\n\n" +
|
||||
"Where there is no room for a second copy and the old replica is all that\n" +
|
||||
"is in the way — one GPU, every card that fits held — it is stopped at once\n" +
|
||||
"and the new one starts in its place; a model with only that replica is\n" +
|
||||
"unserved while it loads. Where the new one would not fit even then — a\n" +
|
||||
"profile edited to need more than any node has — the restart is given up\n" +
|
||||
"and the old replica keeps serving: llmgw ctl replicas says so, and why.\n\n" +
|
||||
"--in-place stops each replica first, whatever room there is.\n\n" +
|
||||
" llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
|
||||
" llmgw ctl restart --model qwen2.5-7b\n" +
|
||||
" llmgw ctl restart --all",
|
||||
@@ -33,7 +38,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
if len(args) == 0 && len(models) == 0 && !all {
|
||||
return errors.New("name replicas, --model, or --all")
|
||||
}
|
||||
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
|
||||
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all, InPlace: inPlace})
|
||||
var res adminapi.RestartResult
|
||||
if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
|
||||
return err
|
||||
@@ -44,6 +49,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
}
|
||||
c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
|
||||
c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
|
||||
c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before starting its replacement")
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
+17
-2
@@ -23,6 +23,7 @@ func newDoctorCmd() *cobra.Command {
|
||||
opts doctor.Options
|
||||
asJSON bool
|
||||
verbose bool
|
||||
fabrics []string
|
||||
)
|
||||
cmd := &cobra.Command{
|
||||
Use: "doctor",
|
||||
@@ -38,8 +39,21 @@ that file when the report is not enough to go on.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
doctor.SetConfigLoader(config.LoadAgent)
|
||||
if opts.VLLMVersion == "" {
|
||||
for _, spec := range fabrics {
|
||||
name, hcas, _ := strings.Cut(spec, ":")
|
||||
if name == "" || hcas == "" {
|
||||
return fmt.Errorf("--fabric %q: give it as name:mlx5_0,mlx5_1", spec)
|
||||
}
|
||||
if opts.Fabrics == nil {
|
||||
opts.Fabrics = map[string][]string{}
|
||||
}
|
||||
opts.Fabrics[name] = strings.Split(hcas, ",")
|
||||
}
|
||||
switch opts.VLLMVersion {
|
||||
case "":
|
||||
opts.VLLMVersion = config.DefaultEngines["vllm"]
|
||||
case "none":
|
||||
opts.VLLMVersion = ""
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
@@ -66,7 +80,8 @@ that file when the report is not enough to go on.`,
|
||||
f.StringVar(&opts.ConfigPath, "config", "", "agent configuration (default /etc/llmgw/agent.yaml when present)")
|
||||
f.StringVar(&opts.IBRoot, "ib-root", discovery.SysfsRoot, "sysfs InfiniBand directory")
|
||||
f.StringVar(&opts.NvidiaSMI, "nvidia-smi", "", "path to nvidia-smi")
|
||||
f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs)")
|
||||
f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs; none to skip)")
|
||||
f.StringArrayVar(&fabrics, "fabric", nil, "a fabric about to be declared, name:mlx5_0,mlx5_1, checked as if it were; repeatable")
|
||||
f.BoolVar(&asJSON, "json", false, "write every fact and finding as JSON, to send back")
|
||||
f.BoolVar(&verbose, "verbose", false, "also print the raw nvidia-smi topology")
|
||||
return cmd
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "engine-build <node> <engine> <build>",
|
||||
Short: "Choose which build of an engine a GPU node installs",
|
||||
Long: "A node installs the build of vLLM its hardware runs: CUDA 13 where the\n" +
|
||||
"driver and GPUs allow, else the same version built for CUDA 12.9. This\n" +
|
||||
"chooses instead, as the cluster page's vLLM build column does:\n\n" +
|
||||
" auto what the hardware runs (the default)\n" +
|
||||
" cu13 the CUDA 13 build, from PyPI\n" +
|
||||
" cu129 the CUDA 12.9 build, from vLLM's GitHub release\n" +
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
|
||||
"gets 0.19.1, the newest built for it, by itself.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
body, _ := json.Marshal(adminapi.NodeEngineRequest{Engine: args[1], Build: args[2]})
|
||||
if err := client().post(cmd.Context(), "/api/v1/nodes/"+args[0]+"/engines", body, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s build %s; the node installs it in the background (llmgw ctl nodes shows it)\n",
|
||||
args[0], args[1], args[2])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
@@ -126,6 +127,14 @@ func newEngineCheckCmd() *cobra.Command {
|
||||
}
|
||||
sort.Strings(unknown)
|
||||
|
||||
// Flags accepted is not the same as running where it is placed: a
|
||||
// GPU build whose backend does not load accepts every flag and
|
||||
// computes on the CPU. Asked of the engine here, not of a help.
|
||||
var deviceErr error
|
||||
if fb, ok := eng.(engines.CPUFallback); ok && help == "" {
|
||||
deviceErr = checkDevices(cmd.Context(), name, fb, out)
|
||||
}
|
||||
|
||||
fmt.Fprintf(out, "%s: %d flags checked", name, checked)
|
||||
if skipped > 0 {
|
||||
fmt.Fprintf(out, ", %d commands skipped", skipped)
|
||||
@@ -137,7 +146,7 @@ func newEngineCheckCmd() *cobra.Command {
|
||||
return fmt.Errorf("nothing was checked; install the engine here or pass --help-output")
|
||||
}
|
||||
fmt.Fprintln(out, "every flag llmgw builds is accepted")
|
||||
return nil
|
||||
return deviceErr
|
||||
}
|
||||
fmt.Fprintf(out, "\n%d flags not mentioned by the program they are passed to:\n", len(unknown))
|
||||
for _, f := range unknown {
|
||||
@@ -249,3 +258,22 @@ func isFlagChar(b byte) bool {
|
||||
return b == '-' || b == '_' || (b >= 'a' && b <= 'z') ||
|
||||
(b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9')
|
||||
}
|
||||
|
||||
// checkDevices says which GPUs an engine that can fall back to the CPU sees on
|
||||
// this machine, and fails when the machine has NVIDIA GPUs and it sees none.
|
||||
func checkDevices(ctx context.Context, name string, fb engines.CPUFallback, out io.Writer) error {
|
||||
if _, err := exec.LookPath("nvidia-smi"); err != nil {
|
||||
fmt.Fprintln(out, "no NVIDIA GPU here: which devices it computes on was not checked")
|
||||
return nil
|
||||
}
|
||||
devs, err := fb.Devices(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(devs) == 0 {
|
||||
return fmt.Errorf("this machine has NVIDIA GPUs and %s sees none: it would serve a GPU placement on the CPU. "+
|
||||
"Its GPU backend did not load; ldd on its libggml-cuda.so names the missing library", name)
|
||||
}
|
||||
fmt.Fprintf(out, "computes on: %s\n", strings.Join(devs, "; "))
|
||||
return nil
|
||||
}
|
||||
+54
-29
@@ -2,6 +2,7 @@ package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
@@ -56,52 +57,69 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
|
||||
var model, version, as string
|
||||
|
||||
c := &cobra.Command{
|
||||
Use: "add <url>",
|
||||
Use: "add <url>...",
|
||||
Short: "Download a model into the mirror",
|
||||
Long: "Downloads to a temporary name and renames when complete, so an\n" +
|
||||
"interrupted download never leaves something a catalog could be pointed\n" +
|
||||
"at. Reports the checksum: a mirror is the one place in this system where\n" +
|
||||
"a corrupted file is copied to every node rather than caught once.\n\n" +
|
||||
"Several URLs go into one version, as the files of one model: the shards\n" +
|
||||
"of a safetensors set with its config.json and tokenizer, for vLLM. The\n" +
|
||||
"version is offered to nodes only once it is whole — nothing still\n" +
|
||||
"downloading, every shard its index names present — and its source is\n" +
|
||||
"the version, ending in a slash.\n\n" +
|
||||
"Refuses to replace a version that is already here. Add a new one beside\n" +
|
||||
"it instead — that is what makes the old one still available when the new\n" +
|
||||
"one turns out to be worse.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
source := args[0]
|
||||
file := as
|
||||
if file == "" {
|
||||
file = mirror.LastPathElement(source)
|
||||
}
|
||||
if model == "" || version == "" {
|
||||
return fmt.Errorf("--model and --version are required: they are the layout")
|
||||
}
|
||||
|
||||
e := mirror.Entry{Model: model, Version: version, File: file}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "fetching %s\n", source)
|
||||
|
||||
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
|
||||
if total > 0 {
|
||||
fmt.Fprintf(out, " %s of %s (%d%%)\n",
|
||||
fileSize(done), fileSize(total), done*100/total)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, " %s so far\n", fileSize(done))
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
if as != "" && len(args) > 1 {
|
||||
return fmt.Errorf("--as names one file; with %d URLs each keeps its own name", len(args))
|
||||
}
|
||||
fmt.Fprintf(out, "\n%s/%s/%s %s\n sha256 %s\n",
|
||||
added.Entry.Model, added.Entry.Version, added.Entry.File,
|
||||
fileSize(added.Entry.Bytes), added.SHA256)
|
||||
fmt.Fprintf(out, "\nPoint a model at it with:\n source: %s\n",
|
||||
added.Entry.URL("http://<this-host>:8081"))
|
||||
out := cmd.OutOrStdout()
|
||||
var last mirror.Added
|
||||
for _, source := range args {
|
||||
file := as
|
||||
if file == "" {
|
||||
file = mirror.LastPathElement(source)
|
||||
}
|
||||
e := mirror.Entry{Model: model, Version: version, File: file}
|
||||
fmt.Fprintf(out, "fetching %s\n", source)
|
||||
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
|
||||
if total > 0 {
|
||||
fmt.Fprintf(out, " %s of %s (%d%%)\n",
|
||||
fileSize(done), fileSize(total), done*100/total)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, " %s so far\n", fileSize(done))
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s/%s %s\n sha256 %s\n",
|
||||
added.Entry.Model, added.Entry.Version, added.Entry.File,
|
||||
fileSize(added.Entry.Bytes), added.SHA256)
|
||||
last = added
|
||||
}
|
||||
src := last.Entry.URL("http://<this-host>:8081")
|
||||
if len(args) > 1 {
|
||||
src = strings.TrimSuffix(mirror.Entry{Model: model, Version: version}.URL("http://<this-host>:8081"), "/") + "/"
|
||||
if m, err := mirror.ManifestFor(*dir, model, version); err != nil {
|
||||
fmt.Fprintf(out, "\nNot yet offered to nodes: %v\n", err)
|
||||
} else {
|
||||
fmt.Fprintf(out, "\n%d files, %s, whole.\n", len(m.Files), fileSize(m.Bytes))
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(out, "\nPoint a model at it with:\n source: %s\n", src)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
|
||||
c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
|
||||
c.Flags().StringVar(&as, "as", "", "the filename to save as; taken from the URL when omitted")
|
||||
c.Flags().StringVar(&as, "as", "", "the filename to save as, for one URL; taken from the URL when omitted")
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -405,7 +423,14 @@ func withManifest(root string, next http.Handler) http.Handler {
|
||||
return
|
||||
}
|
||||
m, err := mirror.ManifestFor(root, model, version)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, mirror.ErrIncomplete):
|
||||
// Said, not hidden behind a 404: a node asking now is told why
|
||||
// it cannot have the version yet, and tries again later.
|
||||
w.Header().Set("Retry-After", "60")
|
||||
http.Error(w, err.Error(), http.StatusConflict)
|
||||
return
|
||||
case err != nil:
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
+20
-1
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
@@ -113,13 +114,14 @@ engines are kept.`,
|
||||
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
|
||||
f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)")
|
||||
f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs")
|
||||
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml")
|
||||
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed, instead of the one the agent installs")
|
||||
f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed")
|
||||
f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH")
|
||||
f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband")
|
||||
f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority")
|
||||
f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
|
||||
f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
|
||||
f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
|
||||
f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
|
||||
f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
|
||||
f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp")
|
||||
@@ -148,6 +150,23 @@ func (na *nodeAdd) prepare() error {
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, e := range o.DisableEngines {
|
||||
if !slices.Contains(provision.Installable(), e) {
|
||||
return fmt.Errorf("--disable-engine %s: the engines are %s", e, strings.Join(provision.Installable(), ", "))
|
||||
}
|
||||
}
|
||||
// What the control plane will ask the node to install, so doctor checks
|
||||
// that vLLM rather than its own default. Its overrides for a GPU class
|
||||
// are not known until the node says what it has, and are not applied
|
||||
// here; doctor's verdict on vLLM is a warning either way.
|
||||
if o.VLLMVersion == "" {
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil {
|
||||
o.VLLMVersion = cfg.WantedEngines()["vllm"]
|
||||
if o.VLLMVersion == "" {
|
||||
o.VLLMVersion = "none"
|
||||
}
|
||||
}
|
||||
}
|
||||
o.Version = version.Version
|
||||
if o.Binary == "" {
|
||||
exe, err := os.Executable()
|
||||
|
||||
+18
-6
@@ -5,19 +5,23 @@ import (
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net"
|
||||
"net/http"
|
||||
"os/signal"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"golang.org/x/net/http2"
|
||||
"golang.org/x/net/http2/h2c"
|
||||
@@ -491,13 +495,21 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
// And what each model was measured to serve, for scaling and for
|
||||
// choosing between profiles.
|
||||
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
|
||||
// The engine builds chosen per node on the cluster page.
|
||||
ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
}
|
||||
svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
|
||||
svc.SetEngines(cfg.WantedEngines())
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines())
|
||||
svc.SetEnginesFor(func(n inventory.Node) map[string]string {
|
||||
return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
|
||||
})
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
|
||||
if ch := ctl.EngineChoices(); ch != nil {
|
||||
svc.SetEngineChoices(ch.For)
|
||||
ch.OnChange(svc.Reprovision)
|
||||
}
|
||||
// A standby must refuse agents as well as refusing to place. Commands
|
||||
// travel down the stream the agent opened, so a standby holding one is a
|
||||
// node the leader cannot command — and the leader's placements would fail
|
||||
|
||||
@@ -8,6 +8,7 @@ package config
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -85,6 +86,13 @@ type Server struct {
|
||||
// one leaves that engine to be installed by hand.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
|
||||
// EngineOverrides give some nodes different engines from the rest: a
|
||||
// vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
// all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
// engines over the cluster's, in order, later ones last; an empty version
|
||||
// removes that engine from the nodes it picks.
|
||||
EngineOverrides []EngineOverride `koanf:"engine_overrides"`
|
||||
|
||||
// BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
// Empty means /var/backups/llmgw-server.
|
||||
@@ -548,6 +556,14 @@ func Load(path string) (Server, error) {
|
||||
// Validate rejects configurations that would otherwise fail confusingly at
|
||||
// request time.
|
||||
func (s Server) Validate() error {
|
||||
for i, o := range s.EngineOverrides {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)
|
||||
}
|
||||
if len(o.Engines) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] changes no engine: give it engines, with \"\" for none", i)
|
||||
}
|
||||
}
|
||||
if s.Listen == "" {
|
||||
return fmt.Errorf("listen is empty")
|
||||
}
|
||||
@@ -626,6 +642,58 @@ var DefaultEngines = map[string]string{
|
||||
"vllm": "0.30.0",
|
||||
}
|
||||
|
||||
// EngineOverride gives the nodes it picks different engines. Every selector
|
||||
// set must match; at least one must be set.
|
||||
type EngineOverride struct {
|
||||
// GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
GPUClass string `koanf:"gpu_class"`
|
||||
// Nodes picks nodes by id.
|
||||
Nodes []string `koanf:"nodes"`
|
||||
// Labels picks nodes carrying all of these.
|
||||
Labels map[string]string `koanf:"labels"`
|
||||
// Engines are laid over the cluster's: a version, or "" for none.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
}
|
||||
|
||||
// Picks reports whether the override applies to a node with this id, these
|
||||
// GPU classes and these labels.
|
||||
func (o EngineOverride) Picks(node string, classes []string, labels map[string]string) bool {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return false
|
||||
}
|
||||
if o.GPUClass != "" && !slices.Contains(classes, o.GPUClass) {
|
||||
return false
|
||||
}
|
||||
if len(o.Nodes) > 0 && !slices.Contains(o.Nodes, node) {
|
||||
return false
|
||||
}
|
||||
for k, v := range o.Labels {
|
||||
if labels[k] != v {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// EnginesFor is what one node installs: WantedEngines with every override
|
||||
// that picks it laid over, in order.
|
||||
func (s Server) EnginesFor(node string, classes []string, labels map[string]string) map[string]string {
|
||||
out := s.WantedEngines()
|
||||
for _, o := range s.EngineOverrides {
|
||||
if !o.Picks(node, classes, labels) {
|
||||
continue
|
||||
}
|
||||
for k, v := range o.Engines {
|
||||
if v == "" {
|
||||
delete(out, k)
|
||||
continue
|
||||
}
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// WantedEngines is DefaultEngines with the configuration laid over it.
|
||||
func (s Server) WantedEngines() map[string]string {
|
||||
out := map[string]string{}
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEachNodeGetsTheEnginesItsOverridesSay(t *testing.T) {
|
||||
s := Server{
|
||||
Engines: map[string]string{"vllm": "0.30.0"},
|
||||
EngineOverrides: []EngineOverride{
|
||||
// Volta: CUDA 13 dropped it, CUDA 12 vLLM still runs.
|
||||
{GPUClass: "v100-32g", Engines: map[string]string{"vllm": "0.25.1"}},
|
||||
// Nodes meant for llama.cpp alone.
|
||||
{Labels: map[string]string{"role": "gguf"}, Engines: map[string]string{"vllm": ""}},
|
||||
},
|
||||
}
|
||||
if got := s.EnginesFor("h1", []string{"h100-80g"}, nil); got["vllm"] != "0.30.0" || got["llamacpp"] == "" {
|
||||
t.Errorf("an H100 node: %v", got)
|
||||
}
|
||||
if got := s.EnginesFor("v1", []string{"v100-32g"}, nil); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("a V100 node: %v", got)
|
||||
}
|
||||
got := s.EnginesFor("v2", []string{"v100-32g"}, map[string]string{"role": "gguf"})
|
||||
if _, has := got["vllm"]; has || got["llamacpp"] == "" {
|
||||
t.Errorf("a V100 node for llama.cpp alone, the later override last: %v", got)
|
||||
}
|
||||
|
||||
for _, bad := range []EngineOverride{
|
||||
{Engines: map[string]string{"vllm": ""}},
|
||||
{GPUClass: "v100-32g"},
|
||||
} {
|
||||
s := Server{Listen: ":8080", EngineOverrides: []EngineOverride{bad}}
|
||||
if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "engine_overrides[0]") {
|
||||
t.Errorf("%+v: %v", bad, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ package adminapi
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
@@ -111,6 +112,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/engines", a.setNodeEngine)
|
||||
mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
|
||||
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
|
||||
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
|
||||
@@ -156,6 +158,10 @@ type NodeView struct {
|
||||
// Provision is where installing each wanted engine stands. An engine is in
|
||||
// Engines only once its installation here is ready.
|
||||
Provision []inventory.EngineInstall `json:"provision,omitempty"`
|
||||
// EngineBuilds are the builds chosen for this node, by engine: "auto"
|
||||
// (its hardware decides), a build ("cu129"), or "none". Only on a node
|
||||
// with GPUs, the only kind with a choice to make.
|
||||
EngineBuilds map[string]string `json:"engine_builds,omitempty"`
|
||||
// OS is the operating system and architecture, as the agent read them.
|
||||
OS string `json:"os,omitempty"`
|
||||
// Warnings are facts about the machine that make it slow or unusable
|
||||
@@ -208,10 +214,13 @@ type NodeView struct {
|
||||
|
||||
// GPUView is one card as a node's page shows it.
|
||||
type GPUView struct {
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
NUMA int `json:"numa"`
|
||||
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
// ComputeCapability is what decides which engine builds run on the card:
|
||||
// CUDA 13 builds need 7.5, so a V100 (7.0) runs only older ones.
|
||||
ComputeCapability string `json:"compute_capability,omitempty"`
|
||||
NUMA int `json:"numa"`
|
||||
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
|
||||
// NearestNIC is the adapter closest to the card, and how close.
|
||||
NearestNIC string `json:"nearest_nic,omitempty"`
|
||||
NICPath string `json:"nic_path,omitempty"`
|
||||
@@ -470,6 +479,13 @@ func (a *API) Nodes() []NodeView {
|
||||
}
|
||||
slices.Sort(v.Engines)
|
||||
v.Provision = n.Provision
|
||||
if ch := a.ctl.EngineChoices(); ch != nil && len(n.GPUs) > 0 {
|
||||
chosen := ch.For(n.ID)
|
||||
v.EngineBuilds = map[string]string{"vllm": "auto"}
|
||||
for e, b := range chosen {
|
||||
v.EngineBuilds[e] = b
|
||||
}
|
||||
}
|
||||
if h := n.Host; h.OSID != "" {
|
||||
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
|
||||
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
|
||||
@@ -497,7 +513,8 @@ func (a *API) Nodes() []NodeView {
|
||||
v.Load = loadView(n.Load, time.Now())
|
||||
v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
|
||||
for _, g := range n.GPUs {
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, ComputeCapability: g.ComputeCapability,
|
||||
NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
for hca, rel := range g.NICPaths {
|
||||
if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
|
||||
(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
|
||||
@@ -1238,6 +1255,9 @@ type RestartRequest struct {
|
||||
Replicas []string `json:"replicas,omitempty"`
|
||||
Models []string `json:"models,omitempty"`
|
||||
All bool `json:"all,omitempty"`
|
||||
// InPlace stops each replica before starting its replacement, rather
|
||||
// than starting the replacement beside it first.
|
||||
InPlace bool `json:"in_place,omitempty"`
|
||||
}
|
||||
|
||||
// RestartResult lists the replicas queued.
|
||||
@@ -1261,7 +1281,11 @@ func (a *API) restart(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
|
||||
return
|
||||
}
|
||||
if err := a.ctl.Reconciler().Restart(ids...); err != nil {
|
||||
restart := a.ctl.Reconciler().Restart
|
||||
if req.InPlace {
|
||||
restart = a.ctl.Reconciler().RestartInPlace
|
||||
}
|
||||
if err := restart(ids...); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
@@ -1521,3 +1545,47 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
|
||||
}
|
||||
|
||||
// NodeEngineRequest chooses one engine's build for a node.
|
||||
type NodeEngineRequest struct {
|
||||
Engine string `json:"engine"`
|
||||
// Build is "auto", a build such as "cu129", or "none".
|
||||
Build string `json:"build"`
|
||||
}
|
||||
|
||||
// setNodeEngine chooses an engine's build for a node, and has the node
|
||||
// install it.
|
||||
func (a *API) setNodeEngine(w http.ResponseWriter, r *http.Request) {
|
||||
ch := a.ctl.EngineChoices()
|
||||
if ch == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the choice in")
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if _, ok := a.ctl.Inventory().Get(id); !ok {
|
||||
writeError(w, http.StatusNotFound, "no node "+id)
|
||||
return
|
||||
}
|
||||
var req NodeEngineRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := ch.Set(r.Context(), id, req.Engine, req.Build, actorOf(r)); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, req)
|
||||
}
|
||||
|
||||
// SetNodeEngine chooses an engine's build for a node, for the cluster page.
|
||||
func (a *API) SetNodeEngine(ctx context.Context, node, engine, build, by string) error {
|
||||
ch := a.ctl.EngineChoices()
|
||||
if ch == nil {
|
||||
return errors.New("this control plane has no store to keep the choice in")
|
||||
}
|
||||
if _, ok := a.ctl.Inventory().Get(node); !ok {
|
||||
return errors.New("no node " + node)
|
||||
}
|
||||
return ch.Set(ctx, node, engine, build, by)
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// choiceStore keeps engine choices in memory.
|
||||
type choiceStore struct{ m map[string]map[string]string }
|
||||
|
||||
func (c *choiceStore) NodeEngines(context.Context) (map[string]map[string]string, error) {
|
||||
return c.m, nil
|
||||
}
|
||||
func (c *choiceStore) SetNodeEngine(_ context.Context, node, engine, variant, _ string) error {
|
||||
if c.m[node] == nil {
|
||||
c.m[node] = map[string]string{}
|
||||
}
|
||||
c.m[node][engine] = variant
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestAnEngineBuildIsChosenForANodeAndTheNodeIsTold(t *testing.T) {
|
||||
a, ctl := setup(t, testModels(), gpuNode("gpu07", 8), inventory.Node{ID: "cpu01", State: inventory.Ready})
|
||||
ch := control.NewEngineChoices(t.Context(), &choiceStore{m: map[string]map[string]string{}}, slog.New(slog.DiscardHandler))
|
||||
var told []string
|
||||
ch.OnChange(func(node string) error { told = append(told, node); return nil })
|
||||
ctl.SetEngineChoices(ch)
|
||||
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"cu129"}`, nil); code != http.StatusOK {
|
||||
t.Fatalf("status %d", code)
|
||||
}
|
||||
if len(told) != 1 || told[0] != "gpu07" {
|
||||
t.Errorf("the node was not told: %v", told)
|
||||
}
|
||||
var nodes []NodeView
|
||||
call(t, a, http.MethodGet, "/api/v1/nodes", "", &nodes)
|
||||
for _, n := range nodes {
|
||||
switch n.ID {
|
||||
case "gpu07":
|
||||
if n.EngineBuilds["vllm"] != "cu129" {
|
||||
t.Errorf("gpu07 shows %v", n.EngineBuilds)
|
||||
}
|
||||
case "cpu01":
|
||||
if n.EngineBuilds != nil {
|
||||
t.Errorf("a node with no GPU has no build to choose: %v", n.EngineBuilds)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{`{"engine":"vllm","build":"cu200"}`, `{"engine":"llamacpp","build":"cu129"}`} {
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", bad, nil); code != http.StatusBadRequest {
|
||||
t.Errorf("%s: status %d", bad, code)
|
||||
}
|
||||
}
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"auto"}`, nil); code != http.StatusOK ||
|
||||
ch.For("gpu07")["vllm"] != "" {
|
||||
t.Errorf("auto should leave it to the node again: %d %v", code, ch.For("gpu07"))
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
|
||||
@@ -25,3 +26,57 @@ func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
|
||||
t.Error("uv, a tool for the others, is offered as an engine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetEnginesFor(func(n inventory.Node) map[string]string {
|
||||
if n.ID == "volta" {
|
||||
return map[string]string{"llamacpp": "b1", "vllm": "0.25.1"}
|
||||
}
|
||||
return map[string]string{"llamacpp": "b1", "vllm": "0.30.0"}
|
||||
})
|
||||
reg := &llmgwv1.Register{InstallableEngines: []string{"llamacpp", "vllm"}}
|
||||
want := func(p *llmgwv1.Provision) map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, e := range p.GetEngines() {
|
||||
out[e.GetEngine()] = e.GetVersion()
|
||||
}
|
||||
return out
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, installable(reg))); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("the Volta node: %v", got)
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, installable(reg))); got["vllm"] != "0.30.0" {
|
||||
t.Errorf("the Hopper node: %v", got)
|
||||
}
|
||||
// A node that disabled vLLM does not offer it, and is not asked for it.
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, []string{"llamacpp"})); got["vllm"] != "" {
|
||||
t.Errorf("asked a node for an engine it does not install: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildChosenForANodeIsWhatItIsTold(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetEngines(map[string]string{"llamacpp": "b1", "vllm": "0.30.0"})
|
||||
s.SetEngineChoices(func(node string) map[string]string {
|
||||
return map[string]map[string]string{"gpu07": {"vllm": "cu129"}, "gpu08": {"vllm": "none"}}[node]
|
||||
})
|
||||
can := []string{"llamacpp", "vllm"}
|
||||
variant := func(node string) (string, bool) {
|
||||
for _, e := range s.provisionFor(inventory.Node{ID: node}, can).GetEngines() {
|
||||
if e.GetEngine() == "vllm" {
|
||||
return e.GetVariant(), true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
if v, ok := variant("gpu07"); !ok || v != "cu129" {
|
||||
t.Errorf("gpu07: %q, %v", v, ok)
|
||||
}
|
||||
if _, ok := variant("gpu08"); ok {
|
||||
t.Error("gpu08 was asked for a vLLM it was chosen to run none of")
|
||||
}
|
||||
if v, ok := variant("gpu09"); !ok || v != "" {
|
||||
t.Errorf("gpu09, left to its hardware: %q, %v", v, ok)
|
||||
}
|
||||
}
|
||||
@@ -53,8 +53,13 @@ type Service struct {
|
||||
mu sync.RWMutex
|
||||
conns map[string]*conn
|
||||
|
||||
// engines is what every node is told to install, set once at start.
|
||||
engines map[string]string
|
||||
// enginesFor is what a node is told to install, set once at start: the
|
||||
// cluster's engines, with any override that picks the node laid over.
|
||||
enginesFor func(node inventory.Node) map[string]string
|
||||
// choices are the engine builds chosen per node, and installableBy what
|
||||
// each connected node said it can install, for telling it again.
|
||||
choices func(nodeID string) map[string]string
|
||||
installableBy map[string][]string
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -68,7 +73,37 @@ type Service struct {
|
||||
}
|
||||
|
||||
// SetEngines sets the engines every node installs, by name and version.
|
||||
func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
|
||||
func (s *Service) SetEngines(e map[string]string) {
|
||||
e = maps.Clone(e)
|
||||
s.enginesFor = func(inventory.Node) map[string]string { return e }
|
||||
}
|
||||
|
||||
// SetEnginesFor sets what each node installs, decided per node: a cluster
|
||||
// of two GPU generations needs two vLLM builds, or vLLM on only one.
|
||||
func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
|
||||
|
||||
// SetEngineChoices sets where the builds chosen for each node come from: by
|
||||
// engine, a build ("cu129") or "none". An engine not named is left to the
|
||||
// node's hardware.
|
||||
func (s *Service) SetEngineChoices(f func(nodeID string) map[string]string) { s.choices = f }
|
||||
|
||||
// Reprovision tells a connected node again what to install, after a choice
|
||||
// for it changed; one not connected is told when it next connects.
|
||||
func (s *Service) Reprovision(nodeID string) error {
|
||||
if s.enginesFor == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.RLock()
|
||||
can, ok := s.installableBy[nodeID]
|
||||
s.mu.RUnlock()
|
||||
node, known := s.inv.Get(nodeID)
|
||||
if !ok || !known {
|
||||
return nil
|
||||
}
|
||||
return s.send(nodeID, &llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: s.provisionFor(node, can)},
|
||||
})
|
||||
}
|
||||
|
||||
// conn is one agent's open stream.
|
||||
type conn struct {
|
||||
@@ -179,15 +214,15 @@ func (s *Service) Connect(
|
||||
// What this node should have installed. Sent on every registration, so a
|
||||
// node rebuilt from nothing is told the same as one that has been here a
|
||||
// month, and a changed version reaches every node on its next connect.
|
||||
if s.engines != nil {
|
||||
p := &llmgwv1.Provision{}
|
||||
if s.enginesFor != nil {
|
||||
can := installable(reg)
|
||||
for _, e := range slices.Sorted(maps.Keys(s.engines)) {
|
||||
if !slices.Contains(can, e) {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
|
||||
s.mu.Lock()
|
||||
if s.installableBy == nil {
|
||||
s.installableBy = map[string][]string{}
|
||||
}
|
||||
s.installableBy[nodeID] = can
|
||||
s.mu.Unlock()
|
||||
p := s.provisionFor(node, can)
|
||||
if err := stream.Send(&llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
|
||||
}); err != nil {
|
||||
@@ -550,6 +585,10 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
|
||||
out.GPUs = append(out.GPUs, inventory.GPUSensor{
|
||||
Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
|
||||
PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
|
||||
UtilPercent: g.GetUtilizationPercent(), MemUtilPercent: g.GetMemoryUtilizationPercent(),
|
||||
MemoryUsedMB: g.GetMemoryUsedMb(), MemoryTotalMB: g.GetMemoryTotalMb(),
|
||||
SMClockMHz: g.GetSmClockMhz(), ECCUncorrected: g.GetEccUncorrected(),
|
||||
Wear: wearFrom(g),
|
||||
})
|
||||
}
|
||||
return out
|
||||
@@ -853,6 +892,24 @@ func rate(before, now uint64, secs float64) float64 {
|
||||
return float64(now-before) / secs
|
||||
}
|
||||
|
||||
// provisionFor is what a node is told to install: the engines decided for
|
||||
// it, those it can install.
|
||||
func (s *Service) provisionFor(node inventory.Node, can []string) *llmgwv1.Provision {
|
||||
p := &llmgwv1.Provision{}
|
||||
wants := s.enginesFor(node)
|
||||
var chosen map[string]string
|
||||
if s.choices != nil {
|
||||
chosen = s.choices(node.ID)
|
||||
}
|
||||
for _, e := range slices.Sorted(maps.Keys(wants)) {
|
||||
if !slices.Contains(can, e) || chosen[e] == "none" {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e], Variant: chosen[e]})
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// installable are the engines a node can be asked to install. An agent older
|
||||
// than the list said nothing, and could install llama.cpp alone: asked for
|
||||
// vLLM too, it reported it unsupported, and a cluster half way through an
|
||||
@@ -908,3 +965,22 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
|
||||
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
|
||||
}
|
||||
}
|
||||
|
||||
// wearFrom is a card's wear, when the agent read it.
|
||||
func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
|
||||
if !g.GetHasWear() {
|
||||
return nil
|
||||
}
|
||||
w := &inventory.GPUWear{
|
||||
PCIeReplays: g.GetPcieReplays(), RetiredSBE: g.GetRetiredPagesSbe(), RetiredDBE: g.GetRetiredPagesDbe(),
|
||||
RetirePending: g.GetRetirementPending(), RemappedCorrectable: g.GetRemappedRowsCorrectable(),
|
||||
RemappedUncorrectable: g.GetRemappedRowsUncorrectable(), RemapPending: g.GetRemapPending(),
|
||||
RemapFailure: g.GetRemapFailure(), NVLinkCRC: g.GetNvlinkCrcErrors(), NVLinkReplay: g.GetNvlinkReplayErrors(),
|
||||
NVLinkRecovery: g.GetNvlinkRecoveryErrors(), XIDs: g.GetXidErrors(), LastXID: int(g.GetLastXid()),
|
||||
LastXIDText: g.GetLastXidText(),
|
||||
}
|
||||
if u := g.GetLastXidUnix(); u > 0 {
|
||||
w.LastXIDAt = time.Unix(u, 0)
|
||||
}
|
||||
return w
|
||||
}
|
||||
@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
|
||||
if p.Nodes <= 0 {
|
||||
p.Nodes = 1
|
||||
}
|
||||
// tp on a llama.cpp GPU profile, written before gpus_per_node meant
|
||||
// anything for it, is read as the width it was meant as: N cards of
|
||||
// one node. Refusing it would have dropped a catalog's model on the
|
||||
// upgrade that made the width work. Only both given, disagreeing, is
|
||||
// refused below, as ambiguous.
|
||||
if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
|
||||
if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
|
||||
p.GPUsPerNode, p.TP, p.PP = n, 1, 0
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case p.CPUOnly():
|
||||
// Nothing to divide up: this profile asks for no cards at all.
|
||||
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
|
||||
p.Weight = 1
|
||||
}
|
||||
}
|
||||
// A Hugging Face repository page for a model vLLM or SGLang runs is the
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -333,6 +350,15 @@ func (m Model) validate() error {
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -381,6 +407,22 @@ func (m Model) validate() error {
|
||||
}
|
||||
continue
|
||||
}
|
||||
// llama.cpp on GPUs splits a model across one node's cards by layer,
|
||||
// and how many is gpus_per_node. Not tp: llama.cpp has no tensor
|
||||
// parallelism, and a tp it was given was a width nothing placed — the
|
||||
// replica stayed unplaced, or ran on one card while four were meant.
|
||||
if p.Engine == "llamacpp" {
|
||||
switch {
|
||||
case p.Nodes > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
m.Name, p.Name, p.Nodes)
|
||||
case p.TP > 1 || p.PP > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
|
||||
"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
|
||||
max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
|
||||
}
|
||||
continue
|
||||
}
|
||||
// The arithmetic has to add up, or the engine is handed a shard spec it
|
||||
// cannot satisfy and fails minutes later during model loading.
|
||||
if want := p.GPUsPerNode * p.Nodes; want != p.TP*max(p.PP, 1) {
|
||||
@@ -704,3 +746,21 @@ func (m Model) engineAdapters() []engines.Adapter {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// repositoryPage reports whether source is a Hugging Face repository's page
|
||||
// (https://huggingface.co/org/repo, or one of its tree views) rather than a
|
||||
// file under /resolve/, and the repository's id.
|
||||
func repositoryPage(source string) (string, bool) {
|
||||
for _, host := range []string{"https://huggingface.co/", "http://huggingface.co/", "https://hf.co/"} {
|
||||
rest, ok := strings.CutPrefix(source, host)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
parts := strings.Split(strings.Trim(rest, "/"), "/")
|
||||
if len(parts) < 2 || (len(parts) > 2 && parts[2] == "resolve") {
|
||||
return "", false
|
||||
}
|
||||
return parts[0] + "/" + parts[1], true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
@@ -321,3 +321,67 @@ func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
|
||||
t.Errorf("ordinary tuning refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
|
||||
four := model("big", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", GPUsPerNode: 4})
|
||||
c, err := New([]Model{four})
|
||||
if err != nil {
|
||||
t.Fatalf("gpus_per_node 4 on llama.cpp: %v", err)
|
||||
}
|
||||
m, _ := c.Resolve("big")
|
||||
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
|
||||
t.Errorf("placement was not asked for four GPUs: %+v", pl)
|
||||
}
|
||||
// tp, written before gpus_per_node meant anything for llama.cpp, is read
|
||||
// as the same width rather than refused: refusing dropped the model on
|
||||
// the upgrade that made the width work.
|
||||
old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
|
||||
c, err = New([]Model{old})
|
||||
if err != nil {
|
||||
t.Fatalf("tp 4 on llama.cpp: %v", err)
|
||||
}
|
||||
got, _ := c.Resolve("old")
|
||||
if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
|
||||
t.Errorf("tp 4 should place on four GPUs: %+v", p)
|
||||
}
|
||||
// Both given and disagreeing is ambiguous, and refused; so is more than
|
||||
// one node.
|
||||
for _, p := range []Profile{
|
||||
{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
|
||||
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
|
||||
} {
|
||||
if _, err := New([]Model{model("x", p)}); err == nil {
|
||||
t.Errorf("%s: accepted", p.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
|
||||
// The field: an AWQ model's page given as a vLLM source, fetched as one
|
||||
// file of HTML. vLLM downloads a repository id itself.
|
||||
for _, src := range []string{
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
|
||||
} {
|
||||
m := model("awq")
|
||||
m.Source = src
|
||||
c, err := New([]Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", src, err)
|
||||
}
|
||||
if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
|
||||
t.Errorf("%s became %q", src, got.Source)
|
||||
}
|
||||
}
|
||||
// llama.cpp needs a file, which a page is not.
|
||||
g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
|
||||
g.Source = "https://huggingface.co/Qwen/X-GGUF"
|
||||
if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
|
||||
t.Errorf("a page for llama.cpp: %v", err)
|
||||
}
|
||||
f := model("file")
|
||||
f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{f}); err != nil {
|
||||
t.Errorf("a file under /resolve/ is a source: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -75,15 +75,18 @@ func DefaultConfig() Config {
|
||||
|
||||
// Controller owns the control plane's state.
|
||||
type Controller struct {
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
lead *Leadership
|
||||
disp reconciler.Dispatcher
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
// engineChoices are the engine builds chosen per node; nil without a
|
||||
// store to keep them in.
|
||||
engineChoices *EngineChoices
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
lead *Leadership
|
||||
disp reconciler.Dispatcher
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
|
||||
mu sync.Mutex
|
||||
last string // fingerprint of the last published list
|
||||
@@ -408,3 +411,23 @@ func EngineKeys(secret []byte) func(replicaID string) string {
|
||||
|
||||
// QuotaUse is every capacity quota beside what its scope holds.
|
||||
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
|
||||
|
||||
// SetEngineChoices gives the controller the engine builds chosen per node.
|
||||
func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
|
||||
|
||||
// EngineChoices are the engine builds chosen per node, or nil.
|
||||
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
|
||||
|
||||
// ModelOwners maps each model to the part of the tenancy tree its capacity
|
||||
// counts against; a model owned by nobody is not in it.
|
||||
func (c *Controller) ModelOwners() map[string]string {
|
||||
out := map[string]string{}
|
||||
if cat := c.Catalog(); cat != nil {
|
||||
for _, m := range cat.Models() {
|
||||
if m.Owner != "" {
|
||||
out[m.Name] = m.Owner
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
)
|
||||
|
||||
// EngineChoiceStore is where the engine builds chosen per node are kept.
|
||||
type EngineChoiceStore interface {
|
||||
NodeEngines(ctx context.Context) (map[string]map[string]string, error)
|
||||
SetNodeEngine(ctx context.Context, node, engine, variant, by string) error
|
||||
}
|
||||
|
||||
// EngineChoices are the engine builds chosen for nodes, rather than left to
|
||||
// their hardware: vLLM's CUDA 12.9 build on a node its CUDA 13 build would
|
||||
// also install on, or no vLLM at all. Kept in the store, held here for the
|
||||
// agent service to ask on every registration, and pushed to a node the
|
||||
// moment its choice changes.
|
||||
type EngineChoices struct {
|
||||
st EngineChoiceStore
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.RWMutex
|
||||
byNode map[string]map[string]string
|
||||
notify func(node string) error
|
||||
}
|
||||
|
||||
// NewEngineChoices reads what was chosen before.
|
||||
func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logger) *EngineChoices {
|
||||
c := &EngineChoices{st: st, log: log, byNode: map[string]map[string]string{}}
|
||||
if got, err := st.NodeEngines(ctx); err != nil {
|
||||
log.Error("could not read the engine builds chosen per node; every node installs what its hardware runs", "err", err)
|
||||
} else {
|
||||
c.byNode = got
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// OnChange is called with a node whose choice changed, to tell it.
|
||||
func (c *EngineChoices) OnChange(f func(node string) error) {
|
||||
c.mu.Lock()
|
||||
c.notify = f
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// For is what was chosen for one node, by engine: a build, or "none".
|
||||
func (c *EngineChoices) For(node string) map[string]string {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
return maps.Clone(c.byNode[node])
|
||||
}
|
||||
|
||||
// EngineBuilds are what can be chosen for each engine that has a choice:
|
||||
// "auto" is the node's hardware deciding, "none" no engine at all.
|
||||
func EngineBuilds(engine string) []string {
|
||||
switch engine {
|
||||
case "vllm":
|
||||
return append(append([]string{"auto"}, provision.VLLMVariants...), "none")
|
||||
case "sglang":
|
||||
return []string{"auto", "none"}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Set chooses an engine's build for a node, keeps it, and tells the node.
|
||||
func (c *EngineChoices) Set(ctx context.Context, node, engine, build, by string) error {
|
||||
builds := EngineBuilds(engine)
|
||||
if builds == nil {
|
||||
return fmt.Errorf("%s has no builds to choose between", engine)
|
||||
}
|
||||
if !slices.Contains(builds, build) {
|
||||
return fmt.Errorf("%s has no build %q; choose %s", engine, build, strings.Join(builds, ", "))
|
||||
}
|
||||
variant := build
|
||||
if build == "auto" {
|
||||
variant = ""
|
||||
}
|
||||
if err := c.st.SetNodeEngine(ctx, node, engine, variant, by); err != nil {
|
||||
return err
|
||||
}
|
||||
c.mu.Lock()
|
||||
if variant == "" {
|
||||
delete(c.byNode[node], engine)
|
||||
} else {
|
||||
if c.byNode[node] == nil {
|
||||
c.byNode[node] = map[string]string{}
|
||||
}
|
||||
c.byNode[node][engine] = variant
|
||||
}
|
||||
notify := c.notify
|
||||
c.mu.Unlock()
|
||||
c.log.Info("engine build chosen for a node", "node", node, "engine", engine, "build", build, "by", by)
|
||||
if notify != nil {
|
||||
if err := notify(node); err != nil {
|
||||
// Kept and applied on its next connection; said, since a node that
|
||||
// is not connected is the usual reason.
|
||||
c.log.Warn("the node was not told yet; it is when it next connects", "node", node, "err", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -73,6 +73,18 @@ type Collector struct {
|
||||
gpuMemoryTotal *prometheus.Desc
|
||||
gpuMemoryUsed *prometheus.Desc
|
||||
gpuUtil *prometheus.Desc
|
||||
gpuAssigned *prometheus.Desc
|
||||
gpuPCIeReplay *prometheus.Desc
|
||||
gpuRetired *prometheus.Desc
|
||||
gpuRetirePend *prometheus.Desc
|
||||
gpuRemapped *prometheus.Desc
|
||||
gpuRemapPend *prometheus.Desc
|
||||
gpuRemapFail *prometheus.Desc
|
||||
gpuNVLinkErr *prometheus.Desc
|
||||
gpuXID *prometheus.Desc
|
||||
gpuLastXID *prometheus.Desc
|
||||
gpuMemUtil *prometheus.Desc
|
||||
gpuSMClock *prometheus.Desc
|
||||
gpuTemp *prometheus.Desc
|
||||
gpuMemTemp *prometheus.Desc
|
||||
gpuPower *prometheus.Desc
|
||||
@@ -149,6 +161,33 @@ func New(src Source) *Collector {
|
||||
"How busy a GPU was over the last sample period. Absent when the driver does not "+
|
||||
"report it, which is not the same as idle. Memory used says a model is loaded; "+
|
||||
"this says whether it is doing anything.", gpu, nil),
|
||||
gpuPCIeReplay: prometheus.NewDesc("llmgw_gpu_pcie_replays",
|
||||
"PCIe link retries on a GPU's link since the driver loaded; one that keeps climbing is a riser, slot or cable.", gpu, nil),
|
||||
gpuRetired: prometheus.NewDesc("llmgw_gpu_retired_pages",
|
||||
"Memory pages retired, by cause (sbe, dbe), on GPUs before Ampere.", append(gpu, "cause"), nil),
|
||||
gpuRetirePend: prometheus.NewDesc("llmgw_gpu_retirement_pending",
|
||||
"1 when retired pages wait for a GPU reset to take effect.", gpu, nil),
|
||||
gpuRemapped: prometheus.NewDesc("llmgw_gpu_remapped_rows",
|
||||
"Memory rows remapped, by kind (correctable, uncorrectable), on Ampere and later.", append(gpu, "kind"), nil),
|
||||
gpuRemapPend: prometheus.NewDesc("llmgw_gpu_remap_pending",
|
||||
"1 when remapped rows wait for a GPU reset to take effect.", gpu, nil),
|
||||
gpuRemapFail: prometheus.NewDesc("llmgw_gpu_remap_failure",
|
||||
"1 when a GPU has run out of spare memory rows: it needs replacing.", gpu, nil),
|
||||
gpuNVLinkErr: prometheus.NewDesc("llmgw_gpu_nvlink_errors",
|
||||
"NVLink errors summed over a GPU's links, by kind (crc, replay, recovery).", append(gpu, "kind"), nil),
|
||||
gpuXID: prometheus.NewDesc("llmgw_gpu_xid_errors",
|
||||
"The driver's XID error events for a GPU since its node's agent started.", gpu, nil),
|
||||
gpuLastXID: prometheus.NewDesc("llmgw_gpu_last_xid",
|
||||
"The code of a GPU's last XID error since its agent started; absent when there was none.", gpu, nil),
|
||||
gpuAssigned: prometheus.NewDesc("llmgw_gpu_assigned",
|
||||
"1 for each GPU a replica holds: which model, which replica, and whose quota it counts against. "+
|
||||
"Joined on node and gpu with the other llmgw_gpu_ series, it says what each model's cards are doing.",
|
||||
[]string{"node", "gpu", "model", "replica", "owner"}, nil),
|
||||
gpuMemUtil: prometheus.NewDesc("llmgw_gpu_memory_utilization_percent",
|
||||
"How much of the sample period a GPU's memory was being read or written: "+
|
||||
"decode is bound by it, so a card busy here and idle above is waiting on memory.", gpu, nil),
|
||||
gpuSMClock: prometheus.NewDesc("llmgw_gpu_sm_clock_mhz",
|
||||
"A GPU's streaming multiprocessor clock now; one held low under load is the card being throttled.", gpu, nil),
|
||||
gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
|
||||
"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
|
||||
gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
|
||||
@@ -271,7 +310,9 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
|
||||
c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
|
||||
c.gpuNVLJoined, c.imexUp,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuPCIeReplay, c.gpuRetired, c.gpuRetirePend, c.gpuRemapped, c.gpuRemapPend, c.gpuRemapFail,
|
||||
c.gpuNVLinkErr, c.gpuXID, c.gpuLastXID,
|
||||
c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
|
||||
c.gpuNVLink, c.gpuECC,
|
||||
c.replicaState, c.replicaReady, c.replicaIdle,
|
||||
@@ -287,6 +328,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
now := time.Now()
|
||||
c.collectQuotas(ch)
|
||||
c.collectAssignments(ch)
|
||||
|
||||
loads := map[string]*inventory.Load{}
|
||||
for _, n := range c.src.Nodes() {
|
||||
@@ -388,20 +430,28 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
byClass[class]++
|
||||
labels := []string{n.ID, strconv.Itoa(g.Index), class}
|
||||
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
|
||||
float64(g.MemoryTotalMB)*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
|
||||
float64(g.MemoryUsedMB)*1024*1024, labels...)
|
||||
if g.UtilizationPercent > 0 {
|
||||
// Only when reported. A driver too old to give it would
|
||||
// otherwise show every card at a flat zero, which reads as an
|
||||
// idle fleet rather than a missing measurement.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
|
||||
float64(g.UtilizationPercent), labels...)
|
||||
s, fresh := sensorFor(n.Load, g.Index, now)
|
||||
if fresh && s.Busy() {
|
||||
// What the card is doing now, sampled every heartbeat.
|
||||
c.gpuBusy(ch, s, labels)
|
||||
} else {
|
||||
// An agent too old to sample it: what it read at
|
||||
// registration, which is all there is.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
|
||||
float64(g.MemoryTotalMB)*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
|
||||
float64(g.MemoryUsedMB)*1024*1024, labels...)
|
||||
if g.UtilizationPercent > 0 {
|
||||
// Only when reported. A driver too old to give it would
|
||||
// otherwise show every card at a flat zero, which reads as
|
||||
// an idle fleet rather than a missing measurement.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
|
||||
float64(g.UtilizationPercent), labels...)
|
||||
}
|
||||
}
|
||||
// The sensors' fresh reading when there is one; the temperature
|
||||
// read at registration otherwise, which is all an older agent sends.
|
||||
if s, ok := sensorFor(n.Load, g.Index, now); ok {
|
||||
if fresh {
|
||||
c.gpuSensor(ch, s, labels)
|
||||
} else if g.Health.TemperatureC > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
|
||||
@@ -414,8 +464,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuHealthy, prometheus.GaugeValue, healthy, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuNVLink, prometheus.GaugeValue,
|
||||
float64(len(g.NVLinkPeers)), labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue,
|
||||
float64(g.Health.ECCErrors), labels...)
|
||||
ecc := float64(g.Health.ECCErrors)
|
||||
if fresh && s.Busy() && s.ECCUncorrected >= 0 {
|
||||
ecc = s.ECCUncorrected
|
||||
}
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue, ecc, labels...)
|
||||
}
|
||||
for class, n2 := range byClass {
|
||||
ch <- prometheus.MustNewConstMetric(c.nodeGPUs, prometheus.GaugeValue,
|
||||
@@ -521,6 +574,12 @@ func b2f(b bool) float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
// OwnerSource says whose quota each model counts against, for labelling
|
||||
// what its replicas hold. Asked for by type assertion.
|
||||
type OwnerSource interface {
|
||||
ModelOwners() map[string]string
|
||||
}
|
||||
|
||||
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
|
||||
// assertion, like HealthSource.
|
||||
type QuotaSource interface {
|
||||
@@ -569,6 +628,23 @@ func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor
|
||||
return inventory.GPUSensor{}, false
|
||||
}
|
||||
|
||||
// gpuBusy is how busy a card is, from a sample that carries it.
|
||||
func (c *Collector) gpuBusy(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue, s.MemoryTotalMB*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue, s.MemoryUsedMB*1024*1024, labels...)
|
||||
// Zero is a real reading here — an idle card — and only -1, the driver
|
||||
// not saying, leaves the series out.
|
||||
if s.UtilPercent >= 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue, s.UtilPercent, labels...)
|
||||
}
|
||||
if s.MemUtilPercent >= 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemUtil, prometheus.GaugeValue, s.MemUtilPercent, labels...)
|
||||
}
|
||||
if s.SMClockMHz > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuSMClock, prometheus.GaugeValue, s.SMClockMHz, labels...)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
|
||||
for _, v := range []struct {
|
||||
d *prometheus.Desc
|
||||
@@ -581,4 +657,61 @@ func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor
|
||||
for _, reason := range s.Throttle {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
|
||||
}
|
||||
if w := s.Wear; w != nil {
|
||||
c.gpuWear(ch, w, labels)
|
||||
}
|
||||
}
|
||||
|
||||
// gpuWear is what wears a card out. A count the card does not keep (-1) is
|
||||
// left out, not published as zero.
|
||||
func (c *Collector) gpuWear(ch chan<- prometheus.Metric, w *inventory.GPUWear, labels []string) {
|
||||
gauge := func(d *prometheus.Desc, v float64, extra ...string) {
|
||||
ch <- prometheus.MustNewConstMetric(d, prometheus.GaugeValue, v, append(append([]string{}, labels...), extra...)...)
|
||||
}
|
||||
flag := func(b bool) float64 {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
if w.PCIeReplays >= 0 {
|
||||
gauge(c.gpuPCIeReplay, float64(w.PCIeReplays))
|
||||
}
|
||||
if w.RetiredSBE >= 0 || w.RetiredDBE >= 0 {
|
||||
gauge(c.gpuRetired, float64(max(w.RetiredSBE, 0)), "sbe")
|
||||
gauge(c.gpuRetired, float64(max(w.RetiredDBE, 0)), "dbe")
|
||||
gauge(c.gpuRetirePend, flag(w.RetirePending))
|
||||
}
|
||||
if w.RemappedCorrectable >= 0 || w.RemappedUncorrectable >= 0 {
|
||||
gauge(c.gpuRemapped, float64(max(w.RemappedCorrectable, 0)), "correctable")
|
||||
gauge(c.gpuRemapped, float64(max(w.RemappedUncorrectable, 0)), "uncorrectable")
|
||||
gauge(c.gpuRemapPend, flag(w.RemapPending))
|
||||
gauge(c.gpuRemapFail, flag(w.RemapFailure))
|
||||
}
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkCRC), "crc")
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkReplay), "replay")
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkRecovery), "recovery")
|
||||
gauge(c.gpuXID, float64(w.XIDs))
|
||||
if w.XIDs > 0 {
|
||||
gauge(c.gpuLastXID, float64(w.LastXID))
|
||||
}
|
||||
}
|
||||
|
||||
// collectAssignments says which replica holds each GPU.
|
||||
func (c *Collector) collectAssignments(ch chan<- prometheus.Metric) {
|
||||
var owners map[string]string
|
||||
if o, ok := c.src.(OwnerSource); ok {
|
||||
owners = o.ModelOwners()
|
||||
}
|
||||
for _, r := range c.src.Replicas() {
|
||||
if !r.State.Counts() {
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
for _, g := range sh.GPUIndices {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuAssigned, prometheus.GaugeValue, 1,
|
||||
sh.NodeID, strconv.Itoa(g), r.Model, r.ID, owners[r.Model])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
@@ -39,3 +40,90 @@ func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHowBusyAGPUIsComesFromTheLatestSampleNotRegistration(t *testing.T) {
|
||||
// Discovery read 87% and 70 GB used when the agent came up; the card has
|
||||
// been idle since. The sample says so, zero included.
|
||||
reg := discovery.GPU{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
|
||||
MemoryUsedMB: 70000, UtilizationPercent: 87}
|
||||
n := inventory.Node{ID: "g1", State: inventory.Ready, GPUs: []discovery.GPU{reg},
|
||||
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 35,
|
||||
UtilPercent: 0, MemUtilPercent: 0, MemoryUsedMB: 512, MemoryTotalMB: 81559,
|
||||
SMClockMHz: 345, ECCUncorrected: 2}}}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
|
||||
"llmgw_gpu_memory_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
|
||||
"llmgw_gpu_memory_used_bytes class=h100-80g gpu=0 node=g1 -> 536870912\n",
|
||||
"llmgw_gpu_sm_clock_mhz class=h100-80g gpu=0 node=g1 -> 345\n",
|
||||
"llmgw_gpu_ecc_errors class=h100-80g gpu=0 node=g1 -> 2\n",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s in:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "-> 87\n") {
|
||||
t.Error("the utilization read at registration was published over the sample's")
|
||||
}
|
||||
|
||||
// An agent too old to sample it: registration's figures, and no zero
|
||||
// standing in for a utilization the driver did not give.
|
||||
n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35}}
|
||||
out = gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
if !strings.Contains(out, "llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 87") {
|
||||
t.Errorf("an older agent's registration figure was dropped:\n%s", out)
|
||||
}
|
||||
n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35, UtilPercent: -1, MemUtilPercent: -1, MemoryTotalMB: 81559}}
|
||||
if out := gather(t, fakeSource{nodes: []inventory.Node{n}}); strings.Contains(out, "llmgw_gpu_utilization_percent ") {
|
||||
t.Errorf("a utilization the driver did not report was published:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// ownedSource adds owners to fakeSource.
|
||||
type ownedSource struct{ fakeSource }
|
||||
|
||||
func (ownedSource) ModelOwners() map[string]string { return map[string]string{"chat": "acme/ml"} }
|
||||
|
||||
func TestEachGPUAReplicaHoldsSaysWhoseItIs(t *testing.T) {
|
||||
src := ownedSource{fakeSource{replicas: []reconciler.Replica{
|
||||
{ID: "chat-1", Model: "chat", State: reconciler.Ready, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{0, 1}}}},
|
||||
{ID: "old-1", Model: "old", State: reconciler.Stopped, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}},
|
||||
}}}
|
||||
out := gather(t, src)
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_assigned gpu=0 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
|
||||
"llmgw_gpu_assigned gpu=1 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s in:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "replica=old-1") {
|
||||
t.Error("a stopped replica's GPU was counted as held")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGPUsWearIsPublishedAndWhatItDoesNotKeepIsLeftOut(t *testing.T) {
|
||||
n := inventory.Node{ID: "g1", State: inventory.Ready,
|
||||
GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559}},
|
||||
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 40, MemoryTotalMB: 81559,
|
||||
Wear: &inventory.GPUWear{PCIeReplays: 12, RetiredSBE: -1, RetiredDBE: -1, RemappedCorrectable: 2,
|
||||
RemappedUncorrectable: 1, RemapPending: true, NVLinkCRC: 7, XIDs: 3, LastXID: 79}}}}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_pcie_replays class=h100-80g gpu=0 node=g1 -> 12\n",
|
||||
"llmgw_gpu_remapped_rows class=h100-80g gpu=0 kind=uncorrectable node=g1 -> 1\n",
|
||||
"llmgw_gpu_remap_pending class=h100-80g gpu=0 node=g1 -> 1\n",
|
||||
"llmgw_gpu_nvlink_errors class=h100-80g gpu=0 kind=crc node=g1 -> 7\n",
|
||||
"llmgw_gpu_xid_errors class=h100-80g gpu=0 node=g1 -> 3\n",
|
||||
"llmgw_gpu_last_xid class=h100-80g gpu=0 node=g1 -> 79\n",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s", want)
|
||||
}
|
||||
}
|
||||
// An H100 keeps no retired pages; none are published.
|
||||
if strings.Contains(out, "llmgw_gpu_retired_pages") {
|
||||
t.Errorf("a count the card does not keep was published:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@
|
||||
package inventory
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -319,8 +320,74 @@ type GPUSensor struct {
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerLimitW float64 `json:"power_limit_w,omitempty"`
|
||||
Throttle []string `json:"throttle,omitempty"`
|
||||
// How busy the card is, as of this sample; -1 is not reported. Zero
|
||||
// MemoryTotalMB is an agent too old to send these, whose figures are
|
||||
// the ones it read at registration.
|
||||
UtilPercent float64 `json:"util_percent,omitempty"`
|
||||
MemUtilPercent float64 `json:"mem_util_percent,omitempty"`
|
||||
MemoryUsedMB float64 `json:"memory_used_mb,omitempty"`
|
||||
MemoryTotalMB float64 `json:"memory_total_mb,omitempty"`
|
||||
SMClockMHz float64 `json:"sm_clock_mhz,omitempty"`
|
||||
ECCUncorrected float64 `json:"ecc_uncorrected,omitempty"`
|
||||
// Wear is what wears the card out, read every minute; nil from an agent
|
||||
// that does not read it.
|
||||
Wear *GPUWear `json:"wear,omitempty"`
|
||||
}
|
||||
|
||||
// GPUWear is a card's wear and failures. Counts of -1 are not reported for
|
||||
// it: a card before Ampere has no remapped rows, one after no retired pages.
|
||||
type GPUWear struct {
|
||||
PCIeReplays int64 `json:"pcie_replays"`
|
||||
RetiredSBE int64 `json:"retired_pages_sbe"`
|
||||
RetiredDBE int64 `json:"retired_pages_dbe"`
|
||||
RetirePending bool `json:"retire_pending,omitempty"`
|
||||
RemappedCorrectable int64 `json:"remapped_rows_correctable"`
|
||||
RemappedUncorrectable int64 `json:"remapped_rows_uncorrectable"`
|
||||
RemapPending bool `json:"remap_pending,omitempty"`
|
||||
RemapFailure bool `json:"remap_failure,omitempty"`
|
||||
NVLinkCRC int64 `json:"nvlink_crc_errors"`
|
||||
NVLinkReplay int64 `json:"nvlink_replay_errors"`
|
||||
NVLinkRecovery int64 `json:"nvlink_recovery_errors"`
|
||||
XIDs int64 `json:"xid_errors"`
|
||||
LastXID int `json:"last_xid,omitempty"`
|
||||
LastXIDText string `json:"last_xid_text,omitempty"`
|
||||
LastXIDAt time.Time `json:"last_xid_at,omitempty"`
|
||||
}
|
||||
|
||||
// Problems says, in a few words each, what is wrong with the card; empty
|
||||
// when nothing is.
|
||||
func (w *GPUWear) Problems() []string {
|
||||
if w == nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
if w.RemapFailure {
|
||||
out = append(out, "out of spare memory rows: replace the card")
|
||||
}
|
||||
if w.RemapPending {
|
||||
out = append(out, "memory rows remapped, waiting for a reset")
|
||||
}
|
||||
if w.RetirePending {
|
||||
out = append(out, "memory pages waiting for a reset to be retired")
|
||||
}
|
||||
if w.RemappedUncorrectable > 0 {
|
||||
out = append(out, fmt.Sprintf("%d rows remapped for uncorrectable errors", w.RemappedUncorrectable))
|
||||
}
|
||||
if w.RetiredDBE > 0 {
|
||||
out = append(out, fmt.Sprintf("%d pages retired for double-bit errors", w.RetiredDBE))
|
||||
}
|
||||
if w.XIDs > 0 {
|
||||
out = append(out, fmt.Sprintf("XID %d (%s)", w.LastXID, w.LastXIDText))
|
||||
}
|
||||
if w.NVLinkCRC > 0 || w.NVLinkReplay > 0 || w.NVLinkRecovery > 0 {
|
||||
out = append(out, fmt.Sprintf("NVLink errors: %d CRC, %d replay, %d recovery", w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Busy reports whether the sample carries how busy the card is.
|
||||
func (s GPUSensor) Busy() bool { return s.MemoryTotalMB > 0 }
|
||||
|
||||
// ShardLoad is one engine's use: cores busy, and resident memory.
|
||||
type ShardLoad struct {
|
||||
CPUCores float64
|
||||
|
||||
@@ -255,6 +255,11 @@ type Reconciler struct {
|
||||
// restarted and not yet begun. One is replaced at a time: replacing them
|
||||
// all at once would need the memory of every replica twice over.
|
||||
restartQueue map[string][]string
|
||||
// restartInPlace are the replicas asked to restart by stopping first.
|
||||
restartInPlace map[string]bool
|
||||
// restartRefused says, by replica, why a restart was given up rather
|
||||
// than leave the model without it.
|
||||
restartRefused map[string]string
|
||||
// parked holds models that failed ParkAfter times in a row.
|
||||
parked map[string]parkRecord
|
||||
// extra is how many replicas load has added above a model's base, and
|
||||
@@ -304,6 +309,8 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
|
||||
reservedPorts: map[int]struct{}{},
|
||||
wanted: map[string]time.Time{},
|
||||
restartQueue: map[string][]string{},
|
||||
restartInPlace: map[string]bool{},
|
||||
restartRefused: map[string]string{},
|
||||
parked: map[string]parkRecord{},
|
||||
extra: map[string]int{},
|
||||
busySince: map[string]time.Time{},
|
||||
@@ -729,6 +736,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
r.mu.Unlock()
|
||||
|
||||
desired := r.desiredFor(m)
|
||||
outcome := notTried
|
||||
switch {
|
||||
case len(liveIDs) < desired && isParked:
|
||||
// Not placed. Its failure is on every status the operator reads.
|
||||
@@ -742,7 +750,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// the real error in noise.
|
||||
return
|
||||
}
|
||||
r.scaleUp(ctx, m, avoid)
|
||||
outcome = r.scaleUp(ctx, m, avoid)
|
||||
case len(liveIDs) > desired:
|
||||
r.scaleDown(ctx, liveIDs[desired:])
|
||||
}
|
||||
@@ -781,7 +789,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
}
|
||||
}
|
||||
|
||||
r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
|
||||
r.retire(ctx, m, retiringIDs, readyLive, len(liveIDs), desired, outcome, isParked)
|
||||
}
|
||||
|
||||
// scaleUp places and starts one replica per pass.
|
||||
@@ -789,7 +797,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// One at a time on purpose: placement decisions depend on what is already
|
||||
// allocated, and starting several at once from the same inventory snapshot
|
||||
// would hand them the same GPUs.
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) scaleOutcome {
|
||||
profiles := r.orderedProfiles(m)
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
@@ -813,13 +821,13 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
// the pass that reaps it, and the next pass places this model — which
|
||||
// keeps the stop-then-start ordering in one place rather than two.
|
||||
if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
|
||||
return
|
||||
return makingRoom
|
||||
}
|
||||
if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
|
||||
return
|
||||
return makingRoom
|
||||
}
|
||||
r.notePlacementFailure(m.Name, res.Explain())
|
||||
return
|
||||
return noRoom
|
||||
}
|
||||
if why := r.overLimit(m, res.Placement); why != "" {
|
||||
// Recorded as the placement's answer, so the model reads as not
|
||||
@@ -829,13 +837,26 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
{Profile: res.Placement.Profile.Name, Reason: why}}}
|
||||
r.mu.Unlock()
|
||||
r.notePlacementFailure(m.Name, why)
|
||||
return
|
||||
return overLimit
|
||||
}
|
||||
if err := r.startReplica(ctx, m, res.Placement); err != nil {
|
||||
r.log.Error("starting replica failed", "model", m.Name, "err", err)
|
||||
return noRoom
|
||||
}
|
||||
return placed
|
||||
}
|
||||
|
||||
// scaleOutcome is what one scaleUp did.
|
||||
type scaleOutcome int
|
||||
|
||||
const (
|
||||
notTried scaleOutcome = iota // nothing was asked of placement this pass
|
||||
placed // a replica was started
|
||||
makingRoom // room is being made: a preemption, an idle replica stopping
|
||||
noRoom // placement found nowhere, and nothing is making room
|
||||
overLimit // there is room, but the owner's quota does not allow it
|
||||
)
|
||||
|
||||
func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *placement.Placement) error {
|
||||
prof, ok := m.ProfileByName(pl.Profile.Name)
|
||||
if !ok {
|
||||
|
||||
@@ -114,6 +114,17 @@ type harness struct {
|
||||
inv *inventory.Store
|
||||
disp *fakeDispatcher
|
||||
clock *clock
|
||||
live *catalog.Live
|
||||
}
|
||||
|
||||
// setModels replaces the catalog the reconciler works from, as an edit does.
|
||||
func (h *harness) setModels(t *testing.T, models ...catalog.Model) {
|
||||
t.Helper()
|
||||
cat, err := catalog.New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.live.Store(cat)
|
||||
}
|
||||
|
||||
func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness {
|
||||
@@ -135,7 +146,8 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
|
||||
cfg.UnreachableAfter = 15 * time.Second
|
||||
cfg.LostAfter = 60 * time.Second
|
||||
|
||||
r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
|
||||
live := catalog.NewLive(cat)
|
||||
r := New(cfg, live, inv, disp, slog.New(slog.DiscardHandler))
|
||||
r.SetClock(c.now)
|
||||
// Nodes were just registered, so their LastSeen is wall-clock now, which is
|
||||
// far from the fake clock's epoch. Re-stamp them onto the fake timeline.
|
||||
@@ -150,7 +162,7 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
|
||||
for _, n := range nodes {
|
||||
touchAt(inv, n.ID, c.now())
|
||||
}
|
||||
return &harness{r: r, inv: inv, disp: disp, clock: c}
|
||||
return &harness{r: r, inv: inv, disp: disp, clock: c, live: live}
|
||||
}
|
||||
|
||||
// touchAt rewrites a node's LastSeen onto the fake clock's timeline.
|
||||
@@ -1787,7 +1799,9 @@ func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
|
||||
|
||||
func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
// One node, room for one replica: the replacement can only start once the
|
||||
// old one has gone. Waiting forever would mean never restarting at all.
|
||||
// old one has gone. It is the only thing in the way, so it goes now —
|
||||
// waiting out a grace period first changed nothing but the wait, which
|
||||
// in the field read as a restart that never happened.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
@@ -1796,14 +1810,8 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, _ := replicaByID(h, old); rep.State != Ready {
|
||||
t.Fatalf("stopped before the grace period: %s", rep.State)
|
||||
}
|
||||
|
||||
h.advance(h.r.cfg.DrainGrace + time.Second)
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
|
||||
t.Fatal("still serving after the grace period with no replacement possible")
|
||||
t.Fatal("still serving with no replacement possible beside it")
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
h.r.Reconcile(t.Context())
|
||||
@@ -1815,6 +1823,52 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartWhoseReplacementFitsNowhereIsGivenUp(t *testing.T) {
|
||||
// The profile was edited to need more cards than the node has: stopping
|
||||
// the old replica would leave the model with nothing at all.
|
||||
m := testModel(1)
|
||||
h := newHarness(t, m, gpuNode("1", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
old := h.r.ReadyReplicas()[0].ID
|
||||
|
||||
m.Profiles[0].TP, m.Profiles[0].GPUsPerNode = 8, 8
|
||||
h.setModels(t, m)
|
||||
if err := h.r.Restart(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for range 3 {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.advance(h.r.cfg.DrainGrace + time.Second)
|
||||
}
|
||||
if rep, ok := replicaByID(h, old); !ok || rep.State != Ready {
|
||||
t.Fatalf("the only replica was stopped for a replacement that fits nowhere: %+v", rep)
|
||||
}
|
||||
if st := h.r.RestartState(old); !strings.HasPrefix(st, "refused: ") || !strings.Contains(st, "fits on no node") {
|
||||
t.Errorf("restart state %q should say it was given up, and why", st)
|
||||
}
|
||||
// Asked again, it is tried again.
|
||||
if err := h.r.Restart(old); err != nil || h.r.RestartState(old) != "queued" {
|
||||
t.Errorf("a new restart did not clear the refusal: %q, %v", h.r.RestartState(old), err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartAskedInPlaceStopsFirst(t *testing.T) {
|
||||
// Room beside it, but in place was asked for: an engine that must not
|
||||
// run twice, or memory somebody knows is short.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
old := h.r.ReadyReplicas()[0].ID
|
||||
if err := h.r.RestartInPlace(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
|
||||
t.Fatalf("asked in place, it kept serving beside its replacement: %+v", rep)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
|
||||
h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
|
||||
for i := 0; i < 3; i++ {
|
||||
|
||||
@@ -6,17 +6,34 @@ import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// retire finishes restarting replicas.
|
||||
//
|
||||
// A retiring replica keeps serving until its replacement is ready, and is then
|
||||
// drained like any other. When no replacement could be placed at all —
|
||||
// nowhere else has the memory, which on a small CPU fleet is the usual case —
|
||||
// it is stopped after DrainGrace so its replacement can take its place: a
|
||||
// restart in place, with the model unserved while it loads if it had no other
|
||||
// replica. Said in the log, since that is the one case with an outage.
|
||||
func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
|
||||
// drained like any other. When no replacement could be placed beside it, what
|
||||
// happens depends on what is in the way:
|
||||
//
|
||||
// - the replica itself — one node, or every card that fits held: it is
|
||||
// stopped now, and its replacement takes its place. Waiting would change
|
||||
// nothing; it used to wait DrainGrace anyway, five minutes reading only
|
||||
// "restart replacing", which looked like a restart that never would.
|
||||
// - more than it: the replacement fits nowhere even with it stopped — its
|
||||
// profile was edited to need more than any node has free. Stopping it
|
||||
// would leave the model with nothing, so the restart is given up, and
|
||||
// says why where the replica is shown.
|
||||
// - something being moved out of the way (a preemption, an idle model
|
||||
// stopping): waited for.
|
||||
// - nothing tried this pass (a backoff): the old rule, stopped after
|
||||
// DrainGrace with no replacement in sight.
|
||||
//
|
||||
// A restart asked for in place (ctl restart --in-place) stops first, always.
|
||||
// Each stop is said in the log, since it is the one case with an outage.
|
||||
func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string, readyLive, live, desired int,
|
||||
outcome scaleOutcome, parked bool,
|
||||
) {
|
||||
for _, id := range ids {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -24,6 +41,7 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
|
||||
if ok {
|
||||
since = rep.RetiringAt
|
||||
}
|
||||
inPlace := r.restartInPlace[id]
|
||||
r.mu.Unlock()
|
||||
if !ok {
|
||||
continue
|
||||
@@ -31,15 +49,48 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
|
||||
switch {
|
||||
case readyLive >= desired:
|
||||
r.drainForRestart(ctx, id)
|
||||
case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
|
||||
r.log.Warn("restarting a replica in place: no room for its replacement beside it",
|
||||
case inPlace:
|
||||
r.log.Warn("restarting a replica in place, as asked: stopped before its replacement starts", "replica", id)
|
||||
r.stopForRestart(ctx, id)
|
||||
case live >= desired:
|
||||
// A replacement is placed and loading; it is waited for.
|
||||
case parked:
|
||||
r.refuseRestart(id, "its replacements fail to start; the model is parked (llmgw ctl retry "+m.Name+")")
|
||||
case outcome == noRoom && r.fitsWithout(m, nil, []victim{{ID: id, Model: m.Name}}):
|
||||
r.log.Warn("restarting a replica in place: it is the only thing in its replacement's way",
|
||||
"replica", id, "model", m.Name)
|
||||
r.stopForRestart(ctx, id)
|
||||
case outcome == noRoom || outcome == overLimit:
|
||||
r.refuseRestart(id, "its replacement fits on no node even with this replica stopped "+
|
||||
"(llmgw ctl explain-placement "+m.Name+")")
|
||||
case outcome == notTried && r.now().Sub(since) > r.cfg.DrainGrace:
|
||||
r.log.Warn("restarting a replica in place: no replacement beside it",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
r.drainForRestart(ctx, id)
|
||||
r.stopReplica(ctx, id)
|
||||
r.stopForRestart(ctx, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// stopForRestart drains a replica and stops it at once, for its replacement
|
||||
// to take its place.
|
||||
func (r *Reconciler) stopForRestart(ctx context.Context, id string) {
|
||||
r.drainForRestart(ctx, id)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
|
||||
// refuseRestart gives up a replica's restart, leaving it serving, and keeps
|
||||
// why for whoever looks at the replica.
|
||||
func (r *Reconciler) refuseRestart(id, why string) {
|
||||
r.mu.Lock()
|
||||
if rep, ok := r.replicas[id]; ok {
|
||||
rep.RetiringAt = time.Time{}
|
||||
}
|
||||
delete(r.restartInPlace, id)
|
||||
r.restartRefused[id] = why
|
||||
r.mu.Unlock()
|
||||
r.log.Warn("not restarting a replica: it keeps serving", "replica", id, "because", why)
|
||||
}
|
||||
|
||||
func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -48,6 +99,7 @@ func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
|
||||
rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
|
||||
shards = slices.Clone(rep.Shards)
|
||||
}
|
||||
delete(r.restartInPlace, id)
|
||||
r.mu.Unlock()
|
||||
for _, s := range shards {
|
||||
if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
|
||||
@@ -78,7 +130,14 @@ var ErrNoSuchReplica = errors.New("no ready replica with that id")
|
||||
// Restart queues replicas to be replaced, each by a new one started beside it
|
||||
// before it is stopped — to pick up an edited profile, an engine key, a new
|
||||
// engine version. Replicas of one model are replaced one at a time.
|
||||
func (r *Reconciler) Restart(ids ...string) error {
|
||||
func (r *Reconciler) Restart(ids ...string) error { return r.restart(false, ids) }
|
||||
|
||||
// RestartInPlace queues replicas to be replaced by stopping each first and
|
||||
// then starting its replacement: for when there is knowingly no room beside
|
||||
// it, and the outage while it loads is accepted.
|
||||
func (r *Reconciler) RestartInPlace(ids ...string) error { return r.restart(true, ids) }
|
||||
|
||||
func (r *Reconciler) restart(inPlace bool, ids []string) error {
|
||||
r.mu.Lock()
|
||||
for _, id := range ids {
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -86,6 +145,10 @@ func (r *Reconciler) Restart(ids ...string) error {
|
||||
r.mu.Unlock()
|
||||
return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
|
||||
}
|
||||
delete(r.restartRefused, id)
|
||||
if inPlace {
|
||||
r.restartInPlace[id] = true
|
||||
}
|
||||
if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
|
||||
r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
|
||||
}
|
||||
@@ -95,8 +158,8 @@ func (r *Reconciler) Restart(ids ...string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// RestartState says where a replica's restart stands: "queued", "replacing",
|
||||
// or "" when none was asked for.
|
||||
// RestartState says where a replica's restart stands: "queued",
|
||||
// "replacing", "in place", "refused: why", or "" when none was asked for.
|
||||
func (r *Reconciler) RestartState(id string) string {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
@@ -104,10 +167,14 @@ func (r *Reconciler) RestartState(id string) string {
|
||||
switch {
|
||||
case !ok:
|
||||
return ""
|
||||
case !rep.RetiringAt.IsZero() && r.restartInPlace[id]:
|
||||
return "in place"
|
||||
case !rep.RetiringAt.IsZero():
|
||||
return "replacing"
|
||||
case slices.Contains(r.restartQueue[rep.Model], id):
|
||||
return "queued"
|
||||
case r.restartRefused[id] != "":
|
||||
return "refused: " + r.restartRefused[id]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
+71
-16
@@ -10,6 +10,7 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
@@ -86,10 +87,12 @@ type Facts struct {
|
||||
|
||||
// What installing vLLM here would meet: the version this llmgw installs,
|
||||
// where, the space there, and whether it already has.
|
||||
VLLMVersion string `json:"vllm_version,omitempty"`
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
VLLMVersion string `json:"vllm_version,omitempty"`
|
||||
// DisabledEngines are those agent.yaml says this node never runs.
|
||||
DisabledEngines []string `json:"disabled_engines,omitempty"`
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
}
|
||||
|
||||
// Modules checked, and why each matters.
|
||||
@@ -103,6 +106,10 @@ type Options struct {
|
||||
// VLLMVersion is the vLLM this llmgw installs; its prerequisites are
|
||||
// checked when set.
|
||||
VLLMVersion string
|
||||
// Fabrics are fabrics about to be declared, by name, with their
|
||||
// adapters: what `node add --fabric` will write into agent.yaml, which
|
||||
// does not exist yet when doctor runs before it.
|
||||
Fabrics map[string][]string
|
||||
}
|
||||
|
||||
// Collect reads the node.
|
||||
@@ -183,6 +190,14 @@ func Collect(ctx context.Context, o Options) Facts {
|
||||
}
|
||||
|
||||
f.readConfig(o)
|
||||
if len(o.Fabrics) > 0 {
|
||||
declared := map[string][]string{}
|
||||
for _, c := range f.Fabrics {
|
||||
declared[c.Fabric] = c.Declared
|
||||
}
|
||||
maps.Copy(declared, o.Fabrics)
|
||||
f.Fabrics = discovery.CheckFabrics(declared, f.HCAs)
|
||||
}
|
||||
f.Host.MinComputeCapability = discovery.MinComputeCapability(f.GPUs)
|
||||
if f.VLLMVersion = o.VLLMVersion; f.VLLMVersion != "" {
|
||||
if f.EnginesDir == "" {
|
||||
@@ -238,6 +253,7 @@ func (f *Facts) readConfig(o Options) {
|
||||
_, err = net.InterfaceByName(f.MgmtIface)
|
||||
f.IfaceExists = err == nil
|
||||
f.EnginesDir = cfg.EnginesDir
|
||||
f.DisabledEngines = cfg.Engines.Disabled
|
||||
f.LaunchMode = cfg.Engines.VLLM.LaunchMode
|
||||
if f.LaunchMode == "" {
|
||||
f.LaunchMode = "mp"
|
||||
@@ -419,8 +435,15 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
switch {
|
||||
case p.Active() && p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID < 0:
|
||||
add(Fail, "fabric", where+" is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang",
|
||||
"give the interface an IPv4 address, or check the adapter's RoCE mode")
|
||||
// Fatal only for a port a declared fabric uses. An adapter
|
||||
// nothing is configured to use — a RoCE NIC on a node added
|
||||
// for single-node inference — kept nodes out for a fabric
|
||||
// nobody asked for.
|
||||
msg := where + " is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang"
|
||||
if declared[h.Name] == "" {
|
||||
msg += "; no declared fabric uses it, so nothing will until one does"
|
||||
}
|
||||
add(level, "fabric", msg, "give the interface an IPv4 address, or check the adapter's RoCE mode")
|
||||
case p.Active():
|
||||
on := ""
|
||||
switch {
|
||||
@@ -430,17 +453,24 @@ func Evaluate(f Facts) []Finding {
|
||||
on = ", network " + p.Network
|
||||
}
|
||||
add(OK, "fabric", fmt.Sprintf("%s %s at %d Gb/s (%s)%s", where, p.State, p.RateGbps, p.LinkLayer, on), "")
|
||||
// Questions about which fabric a port is on matter to the
|
||||
// fabrics that use it; for a port none does they are noted,
|
||||
// not warned of.
|
||||
quiet := Warn
|
||||
if declared[h.Name] == "" {
|
||||
quiet = Info
|
||||
}
|
||||
if p.LinkLayer == discovery.LinkInfiniBand && p.SM == nil {
|
||||
why := f.SMErrors[where]
|
||||
fix := ""
|
||||
if strings.Contains(why, "infiniband-diags") {
|
||||
fix = "install infiniband-diags"
|
||||
}
|
||||
add(Warn, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
|
||||
add(quiet, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
|
||||
"; without it, separate fabrics given one name cannot be caught", fix)
|
||||
}
|
||||
if p.LinkLayer == discovery.LinkInfiniBand && p.SMLID == 0 {
|
||||
add(Warn, "fabric", where+" is active but reports no subnet manager LID", "")
|
||||
add(quiet, "fabric", where+" is active but reports no subnet manager LID", "")
|
||||
}
|
||||
case strings.EqualFold(p.PhysState, "LinkUp"):
|
||||
add(level, "fabric", where+" has a link but is "+p.State+": no subnet manager has configured it",
|
||||
@@ -479,7 +509,14 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
if len(f.HCAs) > 0 {
|
||||
if !f.Modules["ib_uverbs"] {
|
||||
add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
|
||||
// RDMA is for replicas spanning nodes; a node that declares no
|
||||
// fabric runs none, and is a node without it.
|
||||
if len(declared) > 0 {
|
||||
add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
|
||||
} else {
|
||||
add(Warn, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA, which a fabric declared "+
|
||||
"later would need; this node runs single-node replicas without it", "modprobe ib_uverbs")
|
||||
}
|
||||
}
|
||||
if len(f.GPUs) > 0 && !f.Modules["nvidia_peermem"] && !f.Modules["nv_peer_mem"] {
|
||||
add(Warn, "fabric", "nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory",
|
||||
@@ -525,7 +562,9 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
}
|
||||
// --- engines ---
|
||||
if f.VLLMVersion != "" {
|
||||
// An engine disabled on this node: whether it could be installed is not
|
||||
// a question anybody is asking.
|
||||
if f.VLLMVersion != "" && !slices.Contains(f.DisabledEngines, "vllm") {
|
||||
vllmFindings(f, add)
|
||||
}
|
||||
|
||||
@@ -542,28 +581,44 @@ func Evaluate(f Facts) []Finding {
|
||||
// the same decision the agent makes, so the report and the node agree.
|
||||
func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
v := f.VLLMVersion
|
||||
b, err := provision.Recipes["vllm"].Choose(v, f.Host)
|
||||
builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
|
||||
var b provision.Build
|
||||
if err == nil {
|
||||
// The first the agent would try; it moves to the next by itself if
|
||||
// that one does not work here.
|
||||
b = builds[0]
|
||||
if b.Version != "" {
|
||||
v = b.Version
|
||||
}
|
||||
}
|
||||
var nh provision.NotHere
|
||||
switch {
|
||||
case errors.As(err, &nh):
|
||||
add(Info, "engines", "vLLM will not be installed: "+nh.Reason, "")
|
||||
return
|
||||
case err != nil:
|
||||
add(Fail, "engines", "vLLM "+v+" cannot be installed: "+err.Error(),
|
||||
"upgrade the NVIDIA driver, or set the control plane's engines.vllm to a version for this one")
|
||||
// A warning, not a failure: the node is still a node, and serves
|
||||
// llama.cpp. Refusing it here kept older GPU generations out of a
|
||||
// cluster for an engine nobody meant to run on them.
|
||||
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
|
||||
"add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
return
|
||||
case f.VLLMInstalled:
|
||||
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
|
||||
return
|
||||
}
|
||||
if f.EnginesFreeGB > 0 && f.EnginesFreeGB < float64(b.FreeGB) {
|
||||
add(Fail, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB",
|
||||
add(Warn, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB; the node joins without it until there is",
|
||||
v, b.FreeGB, f.EnginesDir, f.EnginesFreeGB),
|
||||
"free space there, or point engines_dir in agent.yaml at a larger disk")
|
||||
return
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
|
||||
note := ""
|
||||
if b.Note != "" {
|
||||
note = "; " + b.Note
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
|
||||
}
|
||||
|
||||
func rank(l Level) int {
|
||||
|
||||
@@ -151,8 +151,21 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
says string
|
||||
}{
|
||||
{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
|
||||
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Fail, "driver 580"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Fail, "9.0 GB"},
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
|
||||
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
|
||||
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
// A V100 has no build of vLLM 0.20 or later; it gets the newest
|
||||
// that has Volta kernels, and doctor says which.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
|
||||
{"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
|
||||
VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
|
||||
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
}
|
||||
@@ -200,3 +213,38 @@ func TestDoctorSaysWhenTheCPUHidesAVX2(t *testing.T) {
|
||||
t.Errorf("ARM: %+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineDisabledOnTheNodeIsNotChecked(t *testing.T) {
|
||||
// The field: a V100 node meant for llama.cpp, refused for a vLLM it was
|
||||
// never going to run.
|
||||
f := Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", DisabledEngines: []string{"vllm"}}
|
||||
for _, fd := range Evaluate(f) {
|
||||
if fd.Area == "engines" {
|
||||
t.Errorf("a disabled vLLM was reported on: %s %q", fd.Level, fd.Message)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdapterNoFabricUsesDoesNotKeepTheNodeOut(t *testing.T) {
|
||||
// The field: a RoCE NIC with no v2 GID on a node added for single-node
|
||||
// inference, no fabric given, and the node refused for it.
|
||||
roce := discovery.HCA{Name: "mlx5_2", Ports: []discovery.Port{{Num: 1, State: "ACTIVE",
|
||||
PhysState: "LinkUp", RateGbps: 100, LinkLayer: discovery.LinkEthernet, RoCEv2GID: -1}}}
|
||||
f := healthy()
|
||||
f.HCAs = append(f.HCAs, roce)
|
||||
f.Modules["ib_uverbs"] = false
|
||||
f.Fabrics = nil // nothing declared
|
||||
fs := Evaluate(f)
|
||||
if fails := levels(fs, Fail); len(fails) > 0 {
|
||||
t.Errorf("an undeclared adapter failed the node: %v", fails)
|
||||
}
|
||||
has(t, fs, Warn, "no declared fabric uses it")
|
||||
has(t, fs, Warn, "runs single-node replicas without it")
|
||||
|
||||
// Declared, the same adapter and module are what the fabric needs.
|
||||
f.Fabrics = discovery.CheckFabrics(map[string][]string{"roce-a": {"mlx5_2"}}, f.HCAs)
|
||||
fs = Evaluate(f)
|
||||
has(t, fs, Fail, "mlx5_2/1 is RoCE with no RoCE v2 GID entry")
|
||||
has(t, fs, Fail, "ib_uverbs is not loaded")
|
||||
}
|
||||
@@ -258,6 +258,41 @@ func (f File) IsWeights() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// companionNames are the files beside safetensors weights that an engine
|
||||
// loading the directory reads: the model's shape, its tokenizer, how it was
|
||||
// quantised, and which shard holds which tensor.
|
||||
var companionNames = map[string]bool{
|
||||
"config.json": true, "generation_config.json": true,
|
||||
"tokenizer.json": true, "tokenizer_config.json": true, "tokenizer.model": true,
|
||||
"special_tokens_map.json": true, "added_tokens.json": true,
|
||||
"vocab.json": true, "vocab.txt": true, "merges.txt": true,
|
||||
"chat_template.jinja": true, "chat_template.json": true,
|
||||
"preprocessor_config.json": true, "processor_config.json": true,
|
||||
"quantize_config.json": true, "quant_config.json": true,
|
||||
}
|
||||
|
||||
// Companions are the files of a repository an engine reads beside a
|
||||
// safetensors set: config.json and the tokenizer, the shard index. Only
|
||||
// those at the top: the directory an engine is given is one level deep.
|
||||
//
|
||||
// Never code (a .py): a model that needs its own Python to load asks the
|
||||
// engine to run what the repository ships, which is a decision for whoever
|
||||
// runs the engine, made on purpose, not a file swept in with the tokenizer.
|
||||
func Companions(files []File) []File {
|
||||
var out []File
|
||||
for _, f := range files {
|
||||
name := f.Path
|
||||
if strings.Contains(name, "/") {
|
||||
continue
|
||||
}
|
||||
if companionNames[name] || strings.HasSuffix(name, ".safetensors.index.json") ||
|
||||
strings.HasSuffix(name, ".tiktoken") {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Repo is a repository's current revision and what is in it.
|
||||
type Repo struct {
|
||||
ID string
|
||||
|
||||
@@ -94,6 +94,13 @@ func addWith(ctx context.Context, client *http.Client, root, source string, e En
|
||||
_ = os.Remove(part)
|
||||
return Added{}, err
|
||||
}
|
||||
// The checksum beside the file, before the file is in place: a manifest
|
||||
// listing the file then always has its checksum to hand on, and a node
|
||||
// can tell a file damaged on the way from the one that was mirrored.
|
||||
if err := writeChecksum(final, sum); err != nil {
|
||||
_ = os.Remove(part)
|
||||
return Added{}, err
|
||||
}
|
||||
if err := os.Rename(part, final); err != nil {
|
||||
return Added{}, err
|
||||
}
|
||||
@@ -146,6 +153,9 @@ func Remove(root string, e Entry) error {
|
||||
if err := os.Remove(path); err != nil {
|
||||
return err
|
||||
}
|
||||
// Its checksum goes with it; left, it would keep the directory from
|
||||
// being empty, and so from going.
|
||||
_ = os.Remove(checksumPath(path))
|
||||
versionDir := filepath.Dir(path)
|
||||
if empty, _ := isEmpty(versionDir); empty {
|
||||
_ = os.Remove(versionDir)
|
||||
|
||||
+102
-2
@@ -16,12 +16,15 @@ package mirror
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
@@ -91,7 +94,7 @@ func List(root string) ([]Entry, error) {
|
||||
continue
|
||||
}
|
||||
for _, f := range files {
|
||||
if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) {
|
||||
if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) || strings.HasPrefix(f.Name(), ".") {
|
||||
continue
|
||||
}
|
||||
info, err := f.Info()
|
||||
@@ -169,6 +172,31 @@ func normalizeURL(u string) string {
|
||||
|
||||
const partSuffix = ".part"
|
||||
|
||||
// checksumPath is where a file's sha256 is kept: beside it, hidden, so no
|
||||
// listing takes it for a file of the model and no name a person gives a
|
||||
// file (which may not start with a dot) can be the same.
|
||||
func checksumPath(file string) string {
|
||||
return filepath.Join(filepath.Dir(file), "."+filepath.Base(file)+".sha256")
|
||||
}
|
||||
|
||||
func writeChecksum(file, sum string) error {
|
||||
tmp := checksumPath(file) + partSuffix
|
||||
if err := os.WriteFile(tmp, []byte(sum+"\n"), 0o644); err != nil { //nolint:gosec // a checksum, for anyone to read
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, checksumPath(file))
|
||||
}
|
||||
|
||||
// readChecksum is a file's recorded sha256, or "" for a file put in place by
|
||||
// hand, which has none.
|
||||
func readChecksum(file string) string {
|
||||
b, err := os.ReadFile(checksumPath(file)) //nolint:gosec // beside a file in the mirror
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
// sha256File is the checksum of a file on disk.
|
||||
//
|
||||
// Reported after a download so an operator has something to compare against
|
||||
@@ -205,8 +233,16 @@ type Manifest struct {
|
||||
type ManifestFile struct {
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
// SHA256 is the file's checksum when the mirror fetched it, for a node
|
||||
// to check what it downloaded; empty for a file put there by hand.
|
||||
SHA256 string `json:"sha256,omitempty"`
|
||||
}
|
||||
|
||||
// ErrIncomplete is a version not yet whole: a download still running, or a
|
||||
// shard its own index names missing. Its manifest is not given out, so no
|
||||
// node starts a model from part of it.
|
||||
var ErrIncomplete = errors.New("this version is not complete")
|
||||
|
||||
// ManifestFor lists the files of one model version.
|
||||
//
|
||||
// Built from the directory rather than from a file written alongside it. A
|
||||
@@ -219,15 +255,79 @@ func ManifestFor(root, model, version string) (Manifest, error) {
|
||||
return Manifest{}, err
|
||||
}
|
||||
m := Manifest{Model: model, Version: version}
|
||||
dir := filepath.Join(root, model, version)
|
||||
for _, e := range entries {
|
||||
if e.Model != model || e.Version != version {
|
||||
continue
|
||||
}
|
||||
m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes})
|
||||
m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes,
|
||||
SHA256: readChecksum(filepath.Join(dir, e.File))})
|
||||
m.Bytes += e.Bytes
|
||||
}
|
||||
if len(m.Files) == 0 {
|
||||
return Manifest{}, fmt.Errorf("%s/%s: %w", model, version, os.ErrNotExist)
|
||||
}
|
||||
if err := complete(dir, m.Files); err != nil {
|
||||
return Manifest{}, fmt.Errorf("%s/%s: %w: %v", model, version, ErrIncomplete, err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// complete says whether a version's files are the whole of it: nothing still
|
||||
// downloading, every shard a safetensors index maps a tensor to, and every
|
||||
// part of a split ("-00002-of-00004").
|
||||
//
|
||||
// From what the files say of each other rather than from a list written when
|
||||
// the fetch began, so a version filled by hand is held to the same.
|
||||
func complete(dir string, files []ManifestFile) error {
|
||||
if parts, _ := filepath.Glob(filepath.Join(dir, "*"+partSuffix)); len(parts) > 0 {
|
||||
var names []string
|
||||
for _, p := range parts {
|
||||
if b := filepath.Base(p); !strings.HasPrefix(b, ".") {
|
||||
names = append(names, strings.TrimSuffix(b, partSuffix))
|
||||
}
|
||||
}
|
||||
if len(names) > 0 {
|
||||
return fmt.Errorf("still downloading %s", strings.Join(names, ", "))
|
||||
}
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, f := range files {
|
||||
have[f.Name] = true
|
||||
}
|
||||
var missing []string
|
||||
want := map[string]bool{}
|
||||
for _, f := range files {
|
||||
if strings.HasSuffix(f.Name, ".safetensors.index.json") {
|
||||
var idx struct {
|
||||
WeightMap map[string]string `json:"weight_map"`
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, f.Name)) //nolint:gosec // a file in the mirror
|
||||
if err != nil || json.Unmarshal(b, &idx) != nil {
|
||||
return fmt.Errorf("%s cannot be read", f.Name)
|
||||
}
|
||||
for _, shard := range idx.WeightMap {
|
||||
want[shard] = true
|
||||
}
|
||||
}
|
||||
if m := splitPart.FindStringSubmatch(f.Name); m != nil {
|
||||
total, _ := strconv.Atoi(m[3])
|
||||
for i := 1; i <= total; i++ {
|
||||
want[fmt.Sprintf("%s-%0*d-of-%s%s", m[1], len(m[2]), i, m[3], m[4])] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
for name := range want {
|
||||
if !have[name] {
|
||||
missing = append(missing, name)
|
||||
}
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
sort.Strings(missing)
|
||||
return fmt.Errorf("missing %s", strings.Join(missing, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// splitPart matches one part of a split: prefix, part, total, extension.
|
||||
var splitPart = regexp.MustCompile(`^(.+)-(\d{5})-of-(\d{5})(\.[A-Za-z0-9]+)$`)
|
||||
@@ -177,3 +177,63 @@ func TestANameThatLeavesTheMirrorIsRefused(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAVersionIsGivenOutOnlyWhole(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "m", "v1")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write := func(name, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
write("config.json", "{}")
|
||||
write("model.safetensors.index.json",
|
||||
`{"weight_map":{"a":"model-00001-of-00002.safetensors","b":"model-00002-of-00002.safetensors"}}`)
|
||||
write("model-00001-of-00002.safetensors", "aa")
|
||||
write("model-00002-of-00002.safetensors.part", "b")
|
||||
|
||||
if _, err := ManifestFor(root, "m", "v1"); !errors.Is(err, ErrIncomplete) || !strings.Contains(err.Error(), "still downloading") {
|
||||
t.Fatalf("a version with a shard downloading was given out: %v", err)
|
||||
}
|
||||
// The download failed and its .part went: the index still names it.
|
||||
_ = os.Remove(filepath.Join(dir, "model-00002-of-00002.safetensors.part"))
|
||||
if _, err := ManifestFor(root, "m", "v1"); !errors.Is(err, ErrIncomplete) ||
|
||||
!strings.Contains(err.Error(), "missing model-00002-of-00002.safetensors") {
|
||||
t.Fatalf("a version missing a shard its index names was given out: %v", err)
|
||||
}
|
||||
write("model-00002-of-00002.safetensors", "bb")
|
||||
m, err := ManifestFor(root, "m", "v1")
|
||||
if err != nil || len(m.Files) != 4 {
|
||||
t.Fatalf("the whole version: %+v, %v", m, err)
|
||||
}
|
||||
|
||||
// A split GGUF with no index is held to its own numbering.
|
||||
gg := filepath.Join(root, "g", "v1")
|
||||
_ = os.MkdirAll(gg, 0o755)
|
||||
_ = os.WriteFile(filepath.Join(gg, "q-00001-of-00003.gguf"), []byte("x"), 0o644)
|
||||
_ = os.WriteFile(filepath.Join(gg, "q-00003-of-00003.gguf"), []byte("x"), 0o644)
|
||||
if _, err := ManifestFor(root, "g", "v1"); !errors.Is(err, ErrIncomplete) || !strings.Contains(err.Error(), "q-00002-of-00003.gguf") {
|
||||
t.Fatalf("a split missing its second part was given out: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMirroredFilesChecksumIsHandedOnAndHiddenFromListings(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
e := Entry{Model: "m", Version: "v1", File: "f.gguf"}
|
||||
added, err := Add(context.Background(), root, serving(t, "weights"), e, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := List(root)
|
||||
if len(entries) != 1 {
|
||||
t.Errorf("the checksum was listed as a file of the model: %+v", entries)
|
||||
}
|
||||
m, err := ManifestFor(root, "m", "v1")
|
||||
if err != nil || m.Files[0].SHA256 != added.SHA256 || added.SHA256 == "" {
|
||||
t.Errorf("manifest %+v, checksum %q: %v", m, added.SHA256, err)
|
||||
}
|
||||
}
|
||||
@@ -18,7 +18,6 @@ import (
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -27,6 +26,8 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
@@ -86,6 +87,15 @@ type Options struct {
|
||||
Releases string
|
||||
// IgnoreChecks goes on when doctor finds a problem.
|
||||
IgnoreChecks bool
|
||||
// DisableEngines are engines this node never installs or advertises,
|
||||
// written into its agent.yaml, and not checked by doctor: vLLM on a node
|
||||
// of cards it does not run on, meant for llama.cpp alone. Narrower than
|
||||
// IgnoreChecks, which also passes over everything else.
|
||||
DisableEngines []string
|
||||
// VLLMVersion is the vLLM the control plane has nodes install, for
|
||||
// doctor to check against; "none" when it has them install none, empty
|
||||
// for doctor's own default.
|
||||
VLLMVersion string
|
||||
// SSHWait is how long an unreachable machine is waited for.
|
||||
SSHWait time.Duration
|
||||
|
||||
@@ -390,6 +400,18 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
|
||||
if r.Opts.IBRoot != "" {
|
||||
args += " --ib-root " + shellQuote(r.Opts.IBRoot)
|
||||
}
|
||||
// The fabrics given, checked as the ones they will be; any adapter they
|
||||
// do not name is checked as one nothing uses.
|
||||
for _, fab := range r.Opts.Fabrics {
|
||||
args += " --fabric " + shellQuote(fab.Name+":"+strings.Join(fab.HCAs, ","))
|
||||
}
|
||||
switch {
|
||||
case slices.Contains(r.Opts.DisableEngines, "vllm"):
|
||||
args += " --vllm none"
|
||||
r.emit(StepCheck, Info, "vLLM is disabled on this node: it joins with llama.cpp alone", "")
|
||||
case r.Opts.VLLMVersion != "":
|
||||
args += " --vllm " + shellQuote(r.Opts.VLLMVersion)
|
||||
}
|
||||
out, err := r.Remote.Run(ctx, args, nil)
|
||||
var rep struct {
|
||||
Findings []doctor.Finding `json:"findings"`
|
||||
@@ -596,8 +618,11 @@ func AgentYAML(o Options, res *Result) string {
|
||||
fmt.Fprintf(&b, " ib_root: %q\n", o.IBRoot)
|
||||
}
|
||||
}
|
||||
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" {
|
||||
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" || len(o.DisableEngines) > 0 {
|
||||
b.WriteString("engines:\n")
|
||||
if len(o.DisableEngines) > 0 {
|
||||
fmt.Fprintf(&b, " disabled: [%s]\n", strings.Join(o.DisableEngines, ", "))
|
||||
}
|
||||
if o.LlamaCppBin != "" {
|
||||
fmt.Fprintf(&b, " llamacpp:\n bin: %q\n", o.LlamaCppBin)
|
||||
}
|
||||
|
||||
@@ -280,3 +280,44 @@ func said(events []Event, level, s string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func TestAnEngineDisabledOnANodeIsWrittenAndNotChecked(t *testing.T) {
|
||||
m := newFake()
|
||||
r, events := runner(t, m, Options{Version: "dev", DisableEngines: []string{"vllm"}, VLLMVersion: "0.30.0"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if yaml := string(m.files["/etc/llmgw/agent.yaml"]); !strings.Contains(yaml, "engines:\n disabled: [vllm]") {
|
||||
t.Errorf("agent.yaml:\n%s", yaml)
|
||||
}
|
||||
if !m.ran("sudo llmgw doctor --json --vllm none") {
|
||||
t.Errorf("doctor was asked about a vLLM this node will never run: %v", m.scripts)
|
||||
}
|
||||
if !said(*events, Info, "joins with llama.cpp alone") {
|
||||
t.Error("the reduced engine set went unsaid")
|
||||
}
|
||||
|
||||
// Not disabled: doctor checks the vLLM the control plane installs, not
|
||||
// its own default.
|
||||
m = newFake()
|
||||
r, _ = runner(t, m, Options{Version: "dev", VLLMVersion: "0.25.1"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !m.ran("sudo llmgw doctor --json --vllm '0.25.1'") && !m.ran("sudo llmgw doctor --json --vllm 0.25.1") {
|
||||
t.Errorf("doctor was not told the control plane's vLLM: %v", m.scripts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheFabricsGivenAreCheckedAsDeclared(t *testing.T) {
|
||||
// agent.yaml is written after doctor runs, so the fabrics on the command
|
||||
// line are handed to doctor: checked strictly, and nothing else is.
|
||||
m := newFake()
|
||||
r, _ := runner(t, m, Options{Version: "dev", Fabrics: []Fabric{{Name: "ib-a", HCAs: []string{"mlx5_0", "mlx5_1"}}}})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !m.ran("--fabric 'ib-a:mlx5_0,mlx5_1'") {
|
||||
t.Errorf("doctor was not told the fabric: %v", m.scripts)
|
||||
}
|
||||
}
|
||||
@@ -196,8 +196,16 @@ func (r *Router) Handler() http.Handler {
|
||||
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
|
||||
pages.SetCapacity(c)
|
||||
}
|
||||
// The engine build each GPU node installs, chosen on the cluster
|
||||
// page.
|
||||
if e, ok := r.clusterView.(web.NodeEngineAdmin); ok {
|
||||
pages.RegisterNodeEngines(mux, e)
|
||||
}
|
||||
pages.RegisterAdminQuotas(mux, r.store, r.acct)
|
||||
// And the report of what that tree actually used.
|
||||
if r.history != nil {
|
||||
pages.SetHistory(r.history)
|
||||
}
|
||||
pages.RegisterAdminUsage(mux, r.store, r.acct)
|
||||
}
|
||||
|
||||
|
||||
@@ -45,6 +45,7 @@ var copyOrder = []string{
|
||||
// derived from this, and a new secret would lock the router out of all
|
||||
// of them at the moment of the move.
|
||||
"cluster_secrets",
|
||||
"node_engines",
|
||||
}
|
||||
|
||||
// notCopied are tables deliberately left behind.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
-- The build of an engine chosen for a node, where it is not left to the
|
||||
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
|
||||
-- or no vLLM at all. A node with no row here installs what its hardware runs.
|
||||
CREATE TABLE node_engines (
|
||||
node_id TEXT NOT NULL,
|
||||
engine TEXT NOT NULL,
|
||||
variant TEXT NOT NULL,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (node_id, engine)
|
||||
);
|
||||
@@ -0,0 +1,11 @@
|
||||
-- The build of an engine chosen for a node, where it is not left to the
|
||||
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
|
||||
-- or no vLLM at all. A node with no row here installs what its hardware runs.
|
||||
CREATE TABLE node_engines (
|
||||
node_id TEXT NOT NULL,
|
||||
engine TEXT NOT NULL,
|
||||
variant TEXT NOT NULL,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (node_id, engine)
|
||||
);
|
||||
@@ -0,0 +1,44 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NodeEngines are the engine builds chosen per node: node id, then engine,
|
||||
// then the build ("cu129") or "none". A node or engine not here is left to
|
||||
// the node's hardware.
|
||||
func (s *DB) NodeEngines(ctx context.Context) (map[string]map[string]string, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `SELECT node_id, engine, variant FROM node_engines`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
out := map[string]map[string]string{}
|
||||
for rows.Next() {
|
||||
var node, engine, variant string
|
||||
if err := rows.Scan(&node, &engine, &variant); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if out[node] == nil {
|
||||
out[node] = map[string]string{}
|
||||
}
|
||||
out[node][engine] = variant
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SetNodeEngine chooses an engine's build for a node; an empty variant
|
||||
// leaves it to the node's hardware again.
|
||||
func (s *DB) SetNodeEngine(ctx context.Context, node, engine, variant, by string) error {
|
||||
if variant == "" {
|
||||
_, err := s.db.ExecContext(ctx, `DELETE FROM node_engines WHERE node_id = ? AND engine = ?`, node, engine)
|
||||
return err
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO node_engines (node_id, engine, variant, updated_at, updated_by) VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT (node_id, engine) DO UPDATE SET variant = excluded.variant,
|
||||
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
|
||||
node, engine, variant, time.Now().UnixMilli(), by)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestAnEngineBuildIsChosenPerNodeAndCanBeLeftToItAgain(t *testing.T) {
|
||||
db := capacityStore(t)
|
||||
ctx := t.Context()
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "cu129", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "none", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := db.NodeEngines(ctx)
|
||||
if err != nil || got["gpu07"]["vllm"] != "none" {
|
||||
t.Fatalf("got %v, %v; the second choice replaces the first", got, err)
|
||||
}
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, _ := db.NodeEngines(ctx); len(got["gpu07"]) != 0 {
|
||||
t.Errorf("left to the node again, still %v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
func TestADuplicateIsRecognisedOnEitherDatabase(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
err error
|
||||
want bool
|
||||
}{
|
||||
{"sqlite", errors.New("constraint failed: UNIQUE constraint failed: users.email (2067)"), true},
|
||||
// What pgx returns, wrapped as database/sql passes it on: "unique"
|
||||
// in lower case, which a check for "UNIQUE" missed.
|
||||
{"postgres", fmt.Errorf("exec: %w", &pgconn.PgError{Code: "23505",
|
||||
Message: `duplicate key value violates unique constraint "users_email_key"`}), true},
|
||||
{"postgres, as text", errors.New(`ERROR: duplicate key value violates unique constraint "users_email_key" (SQLSTATE 23505)`), true},
|
||||
{"another postgres error", &pgconn.PgError{Code: "23502", Message: "null value in column"}, false},
|
||||
{"nothing", nil, false},
|
||||
} {
|
||||
if got := isUniqueViolation(c.err); got != c.want {
|
||||
t.Errorf("%s: %v, want %v", c.name, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
+20
-1
@@ -6,6 +6,8 @@ import (
|
||||
"errors"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
// UserState is where an account is in its life.
|
||||
@@ -66,12 +68,29 @@ func (s *DB) CreateUser(ctx context.Context, u User) error {
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
u.ID, u.Email, u.Name, u.PasswordHash, string(u.State),
|
||||
nullString(u.ProjectID), millis(u.CreatedAt))
|
||||
if err != nil && strings.Contains(err.Error(), "UNIQUE") {
|
||||
if isUniqueViolation(err) {
|
||||
return ErrEmailTaken
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// isUniqueViolation reports whether err is a row refused for a value another
|
||||
// row already has. Each database says it its own way: SQLite in words
|
||||
// ("UNIQUE constraint failed"), PostgreSQL by code, 23505, in a message with
|
||||
// "unique" in lower case — which a check for the upper-case word missed, so a
|
||||
// second sign-up with an address on PostgreSQL was "something went wrong".
|
||||
func isUniqueViolation(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
var pe *pgconn.PgError
|
||||
if errors.As(err, &pe) {
|
||||
return pe.Code == "23505"
|
||||
}
|
||||
m := err.Error()
|
||||
return strings.Contains(m, "UNIQUE constraint failed") || strings.Contains(m, "SQLSTATE 23505")
|
||||
}
|
||||
|
||||
// UserByEmail finds an account by its address.
|
||||
func (s *DB) UserByEmail(ctx context.Context, email string) (User, error) {
|
||||
return s.scanUser(s.db.QueryRowContext(ctx,
|
||||
|
||||
+21
-2
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base32"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
@@ -63,6 +64,9 @@ type AccountPages struct {
|
||||
// hasQuotas is set when /admin/quotas is registered.
|
||||
hasQuotas bool
|
||||
// capacity sets capacity quotas, from the quotas page; nil leaves them off.
|
||||
// history is the metrics store, for what only it knows: the GPU time
|
||||
// each owner held. Nil without one.
|
||||
history History
|
||||
capacity CapacityAdmin
|
||||
// personalOff refuses personal keys; see SetPersonalOff.
|
||||
personalOff bool
|
||||
@@ -169,8 +173,12 @@ func (p *AccountPages) signup(w http.ResponseWriter, r *http.Request) {
|
||||
case errors.Is(err, account.ErrWeakPassword), errors.Is(err, account.ErrInvalidEmail):
|
||||
d.Error = err.Error()
|
||||
default:
|
||||
p.log.Error("signup failed", "err", err)
|
||||
d.Error = "something went wrong; try again"
|
||||
// Nothing of the database's words on the page; a reference that finds
|
||||
// them in the log for whoever runs the gateway.
|
||||
ref := errorRef()
|
||||
p.log.Error("signup failed", "ref", ref, "err", err)
|
||||
d.Error = "something went wrong on our side, and your account was not created. " +
|
||||
"Try again later; if it keeps happening, give an operator this reference: " + ref
|
||||
}
|
||||
render(w, "signup", d)
|
||||
}
|
||||
@@ -638,3 +646,14 @@ func (p *AccountPages) renderWithCSRF(w http.ResponseWriter, r *http.Request,
|
||||
|
||||
// SetLlamaCppVersion tells the pages which llama.cpp build the nodes run.
|
||||
func (p *AccountPages) SetLlamaCppVersion(v string) { p.llamaVersion = v }
|
||||
|
||||
// errorRef is a short reference for an error a page does not explain, logged
|
||||
// beside the error itself.
|
||||
func errorRef() string {
|
||||
var b [4]byte
|
||||
_, _ = rand.Read(b[:])
|
||||
return "err-" + hex.EncodeToString(b[:])
|
||||
}
|
||||
|
||||
// SetHistory gives the pages the metrics store.
|
||||
func (p *AccountPages) SetHistory(h History) { p.history = h }
|
||||
@@ -60,6 +60,9 @@ type repoSet struct {
|
||||
// A shard set goes as one action — a fifteen-shard model is not fifteen
|
||||
// decisions, it is one.
|
||||
Sources []repoSource
|
||||
// Companions are the files fetched with a safetensors set that are not
|
||||
// weights: config.json, the tokenizer, the shard index.
|
||||
Companions []string
|
||||
// Model and Version are what the fetch form starts with. Editable, because
|
||||
// the name a catalog entry uses is the operator's to choose and the
|
||||
// registry's idea of it is only a good guess.
|
||||
@@ -317,6 +320,16 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// The CPU while GPUs sit free is a choice to make on purpose:
|
||||
// made by default, it was a model running at a tenth of the
|
||||
// speed with nothing saying why.
|
||||
if model.Profiles[0].CPUOnly() && r.FormValue("cpu_ok") == "" {
|
||||
if n := freeGPUs(p.fleet); n > 0 {
|
||||
return "", fmt.Errorf("this fleet has %s free and %s was set to run on the CPU; "+
|
||||
"choose a GPU placement, or tick \"run on the CPU anyway\"",
|
||||
plural(n, "GPU", "GPUs"), model.Name)
|
||||
}
|
||||
}
|
||||
// The same check `llmgw ctl model add` makes. This page builds
|
||||
// the source out of a row the mirror reported, so it is right
|
||||
// far more often than something typed — but a file removed
|
||||
@@ -332,8 +345,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
// Placed on the next pass, not now: saying it is serving would
|
||||
// be wrong for however long the weights take to reach a node.
|
||||
return fmt.Sprintf(
|
||||
"%s added to the catalog, %s; it is placed on the next pass",
|
||||
model.Name, plural(model.Scaling.Min, "replica", "replicas")), nil
|
||||
"%s added to the catalog, %s: %s; it is placed on the next pass",
|
||||
model.Name, plural(model.Scaling.Min, "replica", "replicas"), describeServe(model)), nil
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -768,6 +781,17 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
|
||||
rs.Sources = append(rs.Sources,
|
||||
repoSource{URL: got.DownloadURL("", f.Path), Path: f.Path})
|
||||
}
|
||||
// Safetensors are not a model on their own: vLLM loads a directory,
|
||||
// and reads its config and tokenizer before any weight. Fetched as
|
||||
// part of the set, or the set arrives as shards nothing can load.
|
||||
if strings.HasSuffix(strings.ToLower(set.Name), ".safetensors") {
|
||||
for _, f := range hf.Companions(got.Files) {
|
||||
rs.Sources = append(rs.Sources,
|
||||
repoSource{URL: got.DownloadURL("", f.Path), Path: f.Path})
|
||||
rs.Companions = append(rs.Companions, f.Path)
|
||||
rs.Bytes += f.Bytes
|
||||
}
|
||||
}
|
||||
d.RepoSets = append(d.RepoSets, rs)
|
||||
}
|
||||
p.probeKinds(r.Context(), d.RepoSets)
|
||||
|
||||
@@ -710,3 +710,37 @@ func TestARefusedFetchReturnsToItsFormAndAStartedOneToTheList(t *testing.T) {
|
||||
t.Errorf("a started download should be shown where it is followed: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASafetensorsSetIsFetchedWithWhatTheEngineReadsBesideIt(t *testing.T) {
|
||||
// The field: a four-shard AWQ model mirrored as its shards alone, and
|
||||
// vLLM failing at start for want of config.json.
|
||||
reg := &fakeRegistry{repo: hf.Repo{
|
||||
ID: "Qwen/Qwen2.5-32B-Instruct-AWQ", Revision: "abc123def456",
|
||||
Files: []hf.File{
|
||||
{Path: "model-00001-of-00002.safetensors", Bytes: 4_000_000_000},
|
||||
{Path: "model-00002-of-00002.safetensors", Bytes: 3_000_000_000},
|
||||
{Path: "model.safetensors.index.json", Bytes: 90_000},
|
||||
{Path: "config.json", Bytes: 900},
|
||||
{Path: "tokenizer.json", Bytes: 7_000_000},
|
||||
{Path: "tokenizer_config.json", Bytes: 7_000},
|
||||
{Path: "README.md", Bytes: 9_000},
|
||||
{Path: "modeling_custom.py", Bytes: 40_000},
|
||||
{Path: "onnx/config.json", Bytes: 900},
|
||||
},
|
||||
}}
|
||||
body := mirrorGet(t, mirrorPageWith(t, &fakeMirror{}, reg, true),
|
||||
"/admin/mirror?repo=Qwen/Qwen2.5-32B-Instruct-AWQ").Body.String()
|
||||
for _, want := range []string{"config.json", "tokenizer.json", "tokenizer_config.json", "model.safetensors.index.json"} {
|
||||
if !strings.Contains(body, `name="file" value="`+want+`"`) {
|
||||
t.Errorf("%s is not fetched with the set", want)
|
||||
}
|
||||
}
|
||||
for _, not := range []string{"README.md", "modeling_custom.py", "onnx/config.json"} {
|
||||
if strings.Contains(body, `name="file" value="`+not+`"`) {
|
||||
t.Errorf("%s was swept in with the set", not)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(body, "with 4 files the engine reads beside them") {
|
||||
t.Errorf("the companions are not said; got %s", body)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// NodeEngineAdmin chooses an engine's build for a node: "auto", a build
|
||||
// such as "cu129", or "none".
|
||||
type NodeEngineAdmin interface {
|
||||
SetNodeEngine(ctx context.Context, node, engine, build, by string) error
|
||||
}
|
||||
|
||||
// RegisterNodeEngines mounts the choice of engine builds on the cluster
|
||||
// page: for an administrator, with the session's CSRF token, and answered by
|
||||
// going back to the page with what happened.
|
||||
func (p *AccountPages) RegisterNodeEngines(mux *http.ServeMux, a NodeEngineAdmin) {
|
||||
mux.HandleFunc("POST /admin/nodes/{id}/engines", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
sess, _, _ := p.session(r)
|
||||
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
node, engine, build := r.PathValue("id"), r.FormValue("engine"), r.FormValue("build")
|
||||
q := url.Values{}
|
||||
if err := a.SetNodeEngine(r.Context(), node, engine, build, admin.Email); err != nil {
|
||||
p.log.Warn("an engine build was not chosen", "admin", admin.Email, "node", node, "err", err)
|
||||
q.Set("problem", node+": "+err.Error())
|
||||
} else {
|
||||
p.log.Info("an engine build was chosen", "admin", admin.Email, "node", node, "engine", engine, "build", build)
|
||||
q.Set("notice", node+" installs "+engine+" "+buildWords(build)+
|
||||
"; the node is told now, and installs it in the background")
|
||||
}
|
||||
http.Redirect(w, r, "/admin?"+q.Encode(), http.StatusSeeOther)
|
||||
}))
|
||||
}
|
||||
|
||||
func buildWords(b string) string {
|
||||
switch b {
|
||||
case "auto":
|
||||
return "in the build its hardware runs"
|
||||
case "none":
|
||||
return "no longer"
|
||||
}
|
||||
return "built for " + map[string]string{"cu13": "CUDA 13", "cu129": "CUDA 12.9"}[b]
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// buildFleet is one GPU node whose vLLM build was chosen, and one CPU node.
|
||||
type buildFleet struct{ fakeFleet }
|
||||
|
||||
func (buildFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{
|
||||
{ID: "gpu07", State: "ready", GPUTotal: 8, GPUFree: 8,
|
||||
EngineBuilds: map[string]string{"vllm": "cu129"},
|
||||
Provision: []inventory.EngineInstall{{Engine: "vllm", Version: "0.30.0", Variant: "cu129", State: "installing"}}},
|
||||
{ID: "cpu01", State: "ready"},
|
||||
}
|
||||
}
|
||||
|
||||
type recordedChoice struct{ node, engine, build, by string }
|
||||
|
||||
type fakeEngineAdmin struct{ got []recordedChoice }
|
||||
|
||||
func (f *fakeEngineAdmin) SetNodeEngine(_ context.Context, node, engine, build, by string) error {
|
||||
f.got = append(f.got, recordedChoice{node, engine, build, by})
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /admin", p.AdminPage(Dashboard(buildFleet{}, nil, "v1", time.Second, false, NavExtras{}, nil)))
|
||||
fe := &fakeEngineAdmin{}
|
||||
p.RegisterNodeEngines(mux, fe)
|
||||
|
||||
body := mirrorGet(t, mux, "/admin").Body.String()
|
||||
if !strings.Contains(body, `action="/admin/nodes/gpu07/engines"`) ||
|
||||
!strings.Contains(body, `<option value="cu129" selected>CUDA 12.9</option>`) {
|
||||
t.Fatalf("no choice of build for the GPU node, with its current one selected: %s", body)
|
||||
}
|
||||
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
|
||||
t.Error("a node with no GPU was offered a vLLM build")
|
||||
}
|
||||
if !strings.Contains(body, "0.30.0 CUDA 12.9 installing") {
|
||||
t.Error("what is installed is not shown beside the choice")
|
||||
}
|
||||
|
||||
// The read-only page on the admin listener has no session, so no form.
|
||||
_, ro := get(t, Dashboard(buildFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(ro, "/engines\"") {
|
||||
t.Error("the admin listener's page offers a change it cannot sign")
|
||||
}
|
||||
|
||||
w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"tok"}, "engine": {"vllm"}, "build": {"none"}})
|
||||
if w.Code != http.StatusSeeOther || len(fe.got) != 1 || fe.got[0] != (recordedChoice{"gpu07", "vllm", "none", "a@example.com"}) {
|
||||
t.Fatalf("status %d, chose %+v", w.Code, fe.got)
|
||||
}
|
||||
if loc := w.Header().Get("Location"); !strings.Contains(loc, "notice=") {
|
||||
t.Errorf("redirected to %s, saying nothing", loc)
|
||||
}
|
||||
if w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"bad"}, "engine": {"vllm"}, "build": {"none"}}); w.Code != http.StatusBadRequest || len(fe.got) != 1 {
|
||||
t.Errorf("a forged form was accepted: %d", w.Code)
|
||||
}
|
||||
}
|
||||
+182
-6
@@ -43,8 +43,20 @@ type serveForm struct {
|
||||
Engine string
|
||||
// Profile is the profile name, which doubles as how a node_selector would
|
||||
// be written later. Named for the hardware rather than the model, because
|
||||
// a profile is about where a model runs.
|
||||
// a profile is about where a model runs. Derived from the placement, not
|
||||
// typed: a name is only a name, and a field for it read as though calling
|
||||
// a profile "gpu" would put it on one.
|
||||
Profile string
|
||||
// Placements are where it can run: each GPU class the fleet has, and for
|
||||
// llama.cpp the CPU. One is selected.
|
||||
Placements []placeChoice
|
||||
// GPUs is how many cards a replica takes, when placed on a GPU class.
|
||||
GPUs int
|
||||
// Parallel is how many requests one replica serves at once.
|
||||
Parallel int
|
||||
// FreeGPUs is how many cards no replica holds, across the fleet: what
|
||||
// choosing the CPU leaves unused.
|
||||
FreeGPUs int
|
||||
// Files is how many files the version holds, above one for a set.
|
||||
Files int
|
||||
// MinMemoryGB is the host RAM to ask for, estimated from the weights.
|
||||
@@ -126,6 +138,13 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
|
||||
default:
|
||||
f.Profile = "gpu"
|
||||
}
|
||||
f.GPUs, f.Parallel = 1, defaultParallel
|
||||
f.Placements, f.FreeGPUs = placementsFor(fleet, f.Engine)
|
||||
for _, pc := range f.Placements {
|
||||
if pc.Selected {
|
||||
f.Profile = pc.profile()
|
||||
}
|
||||
}
|
||||
// Whether any node can actually run it. Agents report their installed
|
||||
// engines now, so this is a fact rather than the guess it started as —
|
||||
// which was cards, and would have passed a GPU node with no vLLM on it.
|
||||
@@ -274,12 +293,38 @@ func buildModel(get func(string) string) (catalog.Model, error) {
|
||||
if engine == "" {
|
||||
return m, errors.New("which engine should run it?")
|
||||
}
|
||||
p := catalog.Profile{
|
||||
Name: strings.TrimSpace(get("profile")), Engine: engine,
|
||||
Weight: 1, Priority: 100,
|
||||
p := catalog.Profile{Engine: engine, Weight: 1, Priority: 100}
|
||||
gpus := 1
|
||||
if v := strings.TrimSpace(get("gpus")); v != "" {
|
||||
if gpus, err = positive(v, "the GPUs per replica"); err != nil {
|
||||
return m, err
|
||||
}
|
||||
gpus = max(gpus, 1)
|
||||
}
|
||||
if p.Name == "" {
|
||||
p.Name = engine
|
||||
place := placeChoice{Value: strings.TrimSpace(get("placement"))}
|
||||
switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
|
||||
case onGPU && class != "":
|
||||
// Placement is decided by gpu_class, which is what the selector
|
||||
// sets; the count is gpus_per_node for llama.cpp, which spreads
|
||||
// layers over a node's cards, and tp for the others.
|
||||
p.GPUClass = class
|
||||
if engine == "llamacpp" {
|
||||
p.GPUsPerNode = gpus
|
||||
} else {
|
||||
p.TP = gpus
|
||||
}
|
||||
place.GPUs = gpus
|
||||
case place.Value == "" || place.Value == "cpu":
|
||||
if engine != "llamacpp" {
|
||||
return m, fmt.Errorf("%s runs on GPUs; choose a GPU class to place it on", engine)
|
||||
}
|
||||
place.Value = "cpu"
|
||||
default:
|
||||
return m, fmt.Errorf("%q is not a placement", place.Value)
|
||||
}
|
||||
p.Name = place.profile()
|
||||
if v := strings.TrimSpace(get("profile")); v != "" {
|
||||
p.Name = v
|
||||
}
|
||||
// The pool, not the per-request limit: llama.cpp divides it across its
|
||||
// parallel slots, so the number a client may use is smaller and is what
|
||||
@@ -290,6 +335,15 @@ func buildModel(get func(string) string) (catalog.Model, error) {
|
||||
}
|
||||
p.Args = []string{"--ctx-size", v}
|
||||
}
|
||||
if v := strings.TrimSpace(get("parallel")); v != "" && engine == "llamacpp" {
|
||||
n, err := positive(v, "the parallel requests")
|
||||
if err != nil {
|
||||
return m, err
|
||||
}
|
||||
if n > 0 {
|
||||
p.Args = append(p.Args, "--parallel", strconv.Itoa(n))
|
||||
}
|
||||
}
|
||||
if v := strings.TrimSpace(get("min_memory_gb")); v != "" {
|
||||
if p.MinMemoryGB, err = positive(v, "the memory requirement"); err != nil {
|
||||
return m, err
|
||||
@@ -408,3 +462,125 @@ func modeOf(sc catalog.Scaling) string {
|
||||
// defaultCtxPool is the llama.cpp context pool the serve form starts with.
|
||||
// --parallel divides it among slots; it does not multiply it.
|
||||
const defaultCtxPool = 16384
|
||||
|
||||
// defaultParallel is the requests a llama.cpp replica serves at once when
|
||||
// the form is not changed: the agent's own default.
|
||||
const defaultParallel = 4
|
||||
|
||||
// placeChoice is one place a model can run: a GPU class, or the CPU.
|
||||
type placeChoice struct {
|
||||
// Value is "gpu:<class>" or "cpu", as the form submits it.
|
||||
Value string
|
||||
Label string
|
||||
// GPUs is how many cards a replica takes, for naming the profile.
|
||||
GPUs int
|
||||
Selected bool
|
||||
}
|
||||
|
||||
// profile names the profile for where it runs: the class, with the count when
|
||||
// it is more than one card ("t4", "t4-x4"), or "cpu".
|
||||
func (c placeChoice) profile() string {
|
||||
class, ok := strings.CutPrefix(c.Value, "gpu:")
|
||||
if !ok || class == "" {
|
||||
return "cpu"
|
||||
}
|
||||
if c.GPUs > 1 {
|
||||
return fmt.Sprintf("%s-x%d", class, c.GPUs)
|
||||
}
|
||||
return class
|
||||
}
|
||||
|
||||
// placementsFor lists where engine can run on this fleet, with one selected,
|
||||
// and how many GPUs are free.
|
||||
//
|
||||
// A GPU when the fleet has one: a GGUF model was put on the CPU by this form
|
||||
// on a cluster of GPUs, ran at a tenth of their speed, and nothing on the way
|
||||
// said why — the form had no choice to offer and named the profile "cpu". The
|
||||
// class selected is the one with the most cards free.
|
||||
func placementsFor(fleet Source, engine string) ([]placeChoice, int) {
|
||||
var out []placeChoice
|
||||
free, best := 0, -1
|
||||
if fleet != nil {
|
||||
for _, c := range fleet.Summary().GPUs {
|
||||
n := c.Free()
|
||||
free += n
|
||||
out = append(out, placeChoice{
|
||||
Value: "gpu:" + c.Class,
|
||||
Label: fmt.Sprintf("GPU: %s, %d of %d free", c.Class, n, c.Total),
|
||||
})
|
||||
if best < 0 || n > fleet.Summary().GPUs[best].Free() {
|
||||
best = len(out) - 1
|
||||
}
|
||||
}
|
||||
}
|
||||
if best >= 0 {
|
||||
out[best].Selected = true
|
||||
}
|
||||
if engine == "llamacpp" {
|
||||
label := "CPU only: no GPU, many times slower"
|
||||
if len(out) == 0 {
|
||||
label = "CPU: this fleet has no GPU"
|
||||
}
|
||||
out = append(out, placeChoice{Value: "cpu", Label: label, Selected: best < 0})
|
||||
}
|
||||
return out, free
|
||||
}
|
||||
|
||||
// freeGPUs is how many cards no replica holds across the fleet.
|
||||
func freeGPUs(fleet Source) int {
|
||||
if fleet == nil {
|
||||
return 0
|
||||
}
|
||||
n := 0
|
||||
for _, c := range fleet.Summary().GPUs {
|
||||
n += c.Free()
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// describeServe says, in one line, what a model will run as: the summary
|
||||
// shown before it is added and in the message after.
|
||||
func describeServe(m catalog.Model) string {
|
||||
if len(m.Profiles) == 0 {
|
||||
return ""
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
engine := map[string]string{"llamacpp": "llama.cpp", "vllm": "vLLM", "sglang": "SGLang"}[p.Engine]
|
||||
if engine == "" {
|
||||
engine = p.Engine
|
||||
}
|
||||
where := "on the CPU"
|
||||
if p.GPUClass != "" {
|
||||
n := max(p.GPUsPerNode, p.TP, 1)
|
||||
where = fmt.Sprintf("on %s %s", plural(n, "GPU", "GPUs"), p.GPUClass)
|
||||
if n > 1 {
|
||||
where = fmt.Sprintf("on %d %s GPUs of one node", n, p.GPUClass)
|
||||
}
|
||||
}
|
||||
parts := []string{engine + " " + where}
|
||||
if p.MinMemoryGB > 0 {
|
||||
parts = append(parts, fmt.Sprintf("%d GB of host memory", p.MinMemoryGB))
|
||||
}
|
||||
if p.GPUClass == "" && p.CPUCores > 0 {
|
||||
parts = append(parts, plural(p.CPUCores, "core", "cores"))
|
||||
}
|
||||
pool, parallel := argInt(p.Args, "--ctx-size"), argInt(p.Args, "--parallel")
|
||||
switch {
|
||||
case pool > 0 && parallel > 0:
|
||||
parts = append(parts, fmt.Sprintf("context pool %d over %d parallel requests (%d each)", pool, parallel, pool/parallel))
|
||||
case pool > 0:
|
||||
parts = append(parts, fmt.Sprintf("context pool %d", pool))
|
||||
}
|
||||
return strings.Join(parts, ", ")
|
||||
}
|
||||
|
||||
// argInt is the number after flag in args, or 0.
|
||||
func argInt(args []string, flag string) int {
|
||||
for i := 0; i+1 < len(args); i++ {
|
||||
if args[i] == flag {
|
||||
n, _ := strconv.Atoi(args[i+1])
|
||||
return n
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -71,6 +71,8 @@ type fakeFleet struct {
|
||||
gpus int
|
||||
engines []string
|
||||
memoryGB int
|
||||
// used is how many of the cards replicas hold.
|
||||
used int
|
||||
}
|
||||
|
||||
func (f fakeFleet) Nodes() []adminapi.NodeView {
|
||||
@@ -81,7 +83,12 @@ func (f fakeFleet) Nodes() []adminapi.NodeView {
|
||||
func (f fakeFleet) Models() []adminapi.ModelView { return nil }
|
||||
func (f fakeFleet) Replicas(string) []adminapi.ReplicaView { return nil }
|
||||
func (f fakeFleet) Explain(string) adminapi.ExplainView { return adminapi.ExplainView{} }
|
||||
func (f fakeFleet) Summary() adminapi.Summary { return adminapi.Summary{} }
|
||||
func (f fakeFleet) Summary() adminapi.Summary {
|
||||
if f.gpus == 0 {
|
||||
return adminapi.Summary{}
|
||||
}
|
||||
return adminapi.Summary{GPUs: []adminapi.ClassUse{{Class: "t4", Total: f.gpus, Used: f.used}}}
|
||||
}
|
||||
|
||||
func held(model, version, file string) mirror.Status {
|
||||
return mirror.Status{Model: model, Version: version, File: file,
|
||||
@@ -452,3 +459,90 @@ func TestServingOnDemandLoadsWhenAskedFor(t *testing.T) {
|
||||
t.Errorf("on demand was added as %+v", sc)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGGUFOnAFleetWithGPUsIsOfferedTheGPUFirst(t *testing.T) {
|
||||
gguf := []mirror.Status{held("coder", "v1", "coder-q4_k_m.gguf")}
|
||||
body := mirrorGet(t, servePage(t, gguf, &fakeCatalog{}, 4, "llamacpp"),
|
||||
"/admin/mirror?serve=coder:v1").Body.String()
|
||||
// Found in the field: no choice offered, a profile called "cpu", and a
|
||||
// model on the CPU of a GPU cluster at a tenth of the speed.
|
||||
if !strings.Contains(body, `<option value="gpu:t4" selected>GPU: t4, 4 of 4 free</option>`) {
|
||||
t.Errorf("the free GPU class should be offered and chosen; got %s", body)
|
||||
}
|
||||
if !strings.Contains(body, `<option value="cpu">CPU only`) {
|
||||
t.Error("the CPU should still be on offer, unselected")
|
||||
}
|
||||
if !strings.Contains(body, `name="cpu_ok"`) {
|
||||
t.Error("choosing the CPU with GPUs free needs saying so")
|
||||
}
|
||||
|
||||
cpuOnly := mirrorGet(t, servePage(t, gguf, &fakeCatalog{}, 0, "llamacpp"),
|
||||
"/admin/mirror?serve=coder:v1").Body.String()
|
||||
if !strings.Contains(cpuOnly, `<option value="cpu" selected>CPU: this fleet has no GPU</option>`) ||
|
||||
strings.Contains(cpuOnly, `name="cpu_ok"`) {
|
||||
t.Errorf("a fleet without GPUs has only the CPU, and nothing to confirm; got %s", cpuOnly)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheChosenPlacementDecidesWhereItRuns(t *testing.T) {
|
||||
form := func(engine, place, gpus string) url.Values {
|
||||
return url.Values{
|
||||
"name": {"m"}, "source": {"http://x/m"}, "engine": {engine}, "replicas": {"1"},
|
||||
"context_length": {"4096"}, "ctx_size": {"16384"}, "parallel": {"4"},
|
||||
"placement": {place}, "gpus": {gpus}, "min_memory_gb": {"8"},
|
||||
}
|
||||
}
|
||||
m, err := buildModel(form("llamacpp", "gpu:t4", "4").Get)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
if p.GPUClass != "t4" || p.GPUsPerNode != 4 || p.TP > 1 || p.Name != "t4-x4" {
|
||||
t.Errorf("llama.cpp on four t4s: %+v", p)
|
||||
}
|
||||
if got := describeServe(m); got != "llama.cpp on 4 t4 GPUs of one node, 8 GB of host memory, "+
|
||||
"context pool 16384 over 4 parallel requests (4096 each)" {
|
||||
t.Errorf("summary %q", got)
|
||||
}
|
||||
if m, _ := buildModel(form("vllm", "gpu:t4", "2").Get); m.Profiles[0].TP != 2 || m.Profiles[0].GPUsPerNode != 0 {
|
||||
t.Errorf("vLLM's GPU count is tp: %+v", m.Profiles[0])
|
||||
}
|
||||
if m, _ := buildModel(form("llamacpp", "cpu", "1").Get); m.Profiles[0].GPUClass != "" || m.Profiles[0].Name != "cpu" {
|
||||
t.Errorf("the CPU: %+v", m.Profiles[0])
|
||||
}
|
||||
if _, err := buildModel(form("vllm", "cpu", "1").Get); err == nil {
|
||||
t.Error("vLLM on the CPU was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCPUWithGPUsFreeIsAChoiceMadeOnPurpose(t *testing.T) {
|
||||
values := func(extra ...string) url.Values {
|
||||
v := url.Values{
|
||||
"csrf": {"tok"}, "name": {"coder"}, "source": {reachableSource(t)}, "engine": {"llamacpp"},
|
||||
"replicas": {"1"}, "context_length": {"4096"}, "placement": {"cpu"}, "min_memory_gb": {"8"},
|
||||
}
|
||||
for i := 0; i+1 < len(extra); i += 2 {
|
||||
v.Set(extra[i], extra[i+1])
|
||||
}
|
||||
return v
|
||||
}
|
||||
cat := &fakeCatalog{}
|
||||
page := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat,
|
||||
Fleet: fakeFleet{gpus: 4, used: 1, engines: []string{"llamacpp"}}}, true)
|
||||
body := mirrorPost(t, page, "/admin/mirror/serve", values()).Body.String()
|
||||
if len(cat.added) != 0 || !strings.Contains(body, "3 GPUs free") {
|
||||
t.Fatalf("the CPU with three GPUs free went through unasked: %s", body)
|
||||
}
|
||||
mirrorPost(t, page, "/admin/mirror/serve", values("cpu_ok", "yes"))
|
||||
if len(cat.added) != 1 {
|
||||
t.Fatal("ticking the box should add it")
|
||||
}
|
||||
// Every card held: nothing to ask.
|
||||
cat = &fakeCatalog{}
|
||||
full := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat,
|
||||
Fleet: fakeFleet{gpus: 4, used: 4, engines: []string{"llamacpp"}}}, true)
|
||||
mirrorPost(t, full, "/admin/mirror/serve", values())
|
||||
if len(cat.added) != 1 {
|
||||
t.Error("with no GPU free the CPU is the only choice and should not be questioned")
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
@@ -80,6 +81,12 @@ type adminUsageData struct {
|
||||
// chart along the bottom reads as "nobody spent anything".
|
||||
Priced bool
|
||||
|
||||
// GPUHeld and GPUHours are the cards each owner's models held over the
|
||||
// window, from the metrics store; empty without one, or with no owned
|
||||
// model on a GPU.
|
||||
GPUHeld TimeSeries
|
||||
GPUHours []gpuHoursRow
|
||||
|
||||
Error string
|
||||
Version string
|
||||
Now string
|
||||
@@ -135,9 +142,46 @@ func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
|
||||
d.Requests, d.Tokens, d.Spend, d.Rows = usageCharts(rows)
|
||||
d.GPUHeld, d.GPUHours = p.gpuHours(r.Context(), win)
|
||||
render(w, "admin_usage", d)
|
||||
}
|
||||
|
||||
// gpuHoursRow is the GPU time one owner's models held over the window.
|
||||
type gpuHoursRow struct {
|
||||
Owner string
|
||||
Hours float64
|
||||
// Now is how many cards they hold at the end of the window.
|
||||
Now float64
|
||||
}
|
||||
|
||||
// gpuHours is the cards held by each owner's models over the window, drawn
|
||||
// and summed into GPU-hours. From the metrics store, because what a model
|
||||
// holds is placement's, not the request log's: a model nobody asks still
|
||||
// holds its cards. As far back as the store keeps history, which for a
|
||||
// default Prometheus is fifteen days.
|
||||
func (p *AccountPages) gpuHours(ctx context.Context, win usageWindow) (TimeSeries, []gpuHoursRow) {
|
||||
if p.history == nil {
|
||||
return TimeSeries{}, nil
|
||||
}
|
||||
series, err := p.history.RangeQuery(ctx, `sum by (owner) (llmgw_gpu_assigned)`, "owner", win.Span, win.Width)
|
||||
if err != nil || len(series) == 0 {
|
||||
return TimeSeries{}, nil
|
||||
}
|
||||
var rows []gpuHoursRow
|
||||
for i := range series {
|
||||
if series[i].Name == "" {
|
||||
series[i].Name = "no owner"
|
||||
}
|
||||
row := gpuHoursRow{Owner: series[i].Name, Now: series[i].Last()}
|
||||
// Each point is the cards held at that instant, standing for the
|
||||
// step that follows it.
|
||||
row.Hours = series[i].Sum() * win.Width.Hours()
|
||||
rows = append(rows, row)
|
||||
}
|
||||
sort.Slice(rows, func(i, j int) bool { return rows[i].Hours > rows[j].Hours })
|
||||
return LineChart(series, "", 6), rows
|
||||
}
|
||||
|
||||
// seriesOf turns each group's buckets into a line of one measure.
|
||||
//
|
||||
// One pass per measure rather than one query per measure: the buckets are
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -162,3 +163,32 @@ func TestAPersonWithNoAccountIsNamedRatherThanBlank(t *testing.T) {
|
||||
t.Errorf("unowned traffic was left unlabelled; got %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheUsagePageCountsTheGPUTimeEachOwnerHeld(t *testing.T) {
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
// Over a day in 20-minute steps: team:research held 8 cards throughout,
|
||||
// nobody's model 2 for the last three steps.
|
||||
res, none := promq.Series{Name: "team:research"}, promq.Series{Name: ""}
|
||||
for i := range 72 {
|
||||
res.Points = append(res.Points, promq.Point{Value: 8})
|
||||
if i >= 69 {
|
||||
none.Points = append(none.Points, promq.Point{Value: 2})
|
||||
}
|
||||
}
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{`sum by (owner) (llmgw_gpu_assigned)`: {res, none}}}
|
||||
p.SetHistory(h)
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsage(mux, &fakeScopeSeries{}, nil)
|
||||
body := mirrorGet(t, mux, "/admin/usage?window=24h").Body.String()
|
||||
for _, want := range []string{"GPU time, by owner", "<td>team:research</td><td class=\"num\">192.0</td>",
|
||||
"<td>no owner</td><td class=\"num\">2.0</td>"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// Without a metrics store, nothing is said about GPU time.
|
||||
if body := mirrorGet(t, usagePage(t, &fakeScopeSeries{}, nil), "/admin/usage").Body.String(); strings.Contains(body, "GPU time") {
|
||||
t.Error("GPU time shown with no metrics store to read it from")
|
||||
}
|
||||
}
|
||||
@@ -19,13 +19,23 @@ type DashboardData struct {
|
||||
// of a GPU table that would only say there are none.
|
||||
CPU cpuCapacity
|
||||
// Menu is the shared top menu; Nav below is what the cluster page adds.
|
||||
Menu Nav
|
||||
Nodes []adminapi.NodeView
|
||||
Models []adminapi.ModelView
|
||||
Replicas []adminapi.ReplicaView
|
||||
Unplaced []adminapi.ExplainView
|
||||
Usage []store.KeyUsage
|
||||
Summary adminapi.Summary
|
||||
Menu Nav
|
||||
// GPUMap is every card coloured by one figure, and IdleGPUs the replicas
|
||||
// holding cards that are doing nothing; both from the last heartbeats.
|
||||
GPUMap *gpuMap
|
||||
IdleGPUs []idleHolding
|
||||
// Efficiency is what each model gets from its cards, from the metrics
|
||||
// store; nil without one.
|
||||
Efficiency []modelEfficiency
|
||||
// Notice and Problem are what the last change made from this page did,
|
||||
// carried over the redirect back to it.
|
||||
Notice, Problem string
|
||||
Nodes []adminapi.NodeView
|
||||
Models []adminapi.ModelView
|
||||
Replicas []adminapi.ReplicaView
|
||||
Unplaced []adminapi.ExplainView
|
||||
Usage []store.KeyUsage
|
||||
Summary adminapi.Summary
|
||||
// Fabrics is the physical fabrics and what disagrees about them, from a
|
||||
// source that can say.
|
||||
Fabrics *fabrics.Report
|
||||
@@ -170,6 +180,9 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
Now: time.Now().UTC().Format(time.RFC3339),
|
||||
AuthEnabled: usage != nil,
|
||||
}
|
||||
if d.Menu.Admin {
|
||||
d.Notice, d.Problem = r.URL.Query().Get("notice"), r.URL.Query().Get("problem")
|
||||
}
|
||||
if src != nil {
|
||||
d.Nodes = src.Nodes()
|
||||
d.CPU = cpuCapacityOf(d.Nodes)
|
||||
@@ -192,9 +205,17 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
d.AnyFabrics = d.AnyFabrics || len(n.Fabrics) > 0
|
||||
}
|
||||
d.NodePrefix = sharedPrefix(ids)
|
||||
if d.AnyGPUs {
|
||||
d.GPUMap = buildGPUMap(d.Nodes, d.Replicas, r.URL.Query().Get("gpumap"), d.NodePrefix)
|
||||
d.IdleGPUs = idleHoldings(d.Nodes, d.Replicas, d.NodePrefix)
|
||||
}
|
||||
for _, m := range d.Models {
|
||||
d.AnyFallbacks = d.AnyFallbacks || len(m.Fallbacks) > 0
|
||||
}
|
||||
if d.AnyGPUs {
|
||||
d.Fleet.LoadGPU(r.Context(), nav.History, nav.Window)
|
||||
d.Efficiency = loadEfficiency(r.Context(), nav.History)
|
||||
}
|
||||
d.Summary = src.Summary()
|
||||
if fs, ok := src.(FabricSource); ok {
|
||||
r := fs.Fabrics()
|
||||
|
||||
@@ -24,7 +24,10 @@ type FleetCharts struct {
|
||||
Queue TimeSeries
|
||||
KV TimeSeries
|
||||
Ready TimeSeries
|
||||
Window string
|
||||
// GPUBusy and GPUMemory are the fleet's cards by class, for a fleet that
|
||||
// has them.
|
||||
GPUBusy, GPUMemory TimeSeries
|
||||
Window string
|
||||
// Err is set when the metrics store could not be read. Shown as a line of
|
||||
// text where the charts would be: an empty panel reads as a healthy cluster
|
||||
// with nothing happening, which is the opposite of not knowing.
|
||||
@@ -90,3 +93,25 @@ func LoadFleetCharts(ctx context.Context, h History, window time.Duration,
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// LoadGPU adds the fleet's GPUs, by class: how busy on average, and how full
|
||||
// their memory. Asked only of a fleet with GPUs. A failure leaves these two
|
||||
// empty and the rest of the page as it was; they are not what the other
|
||||
// panels' all-or-nothing is about.
|
||||
func (f *FleetCharts) LoadGPU(ctx context.Context, h History, window time.Duration) {
|
||||
if f == nil || h == nil || f.Err != "" {
|
||||
return
|
||||
}
|
||||
if window <= 0 {
|
||||
window = time.Hour
|
||||
}
|
||||
step := window / 60
|
||||
if s, err := h.RangeQuery(ctx, `avg by (class) (llmgw_gpu_utilization_percent) / 100`, "class", window, step); err == nil {
|
||||
f.GPUBusy = LineChart(s, "%", 6)
|
||||
}
|
||||
if s, err := h.RangeQuery(ctx,
|
||||
`sum by (class) (llmgw_gpu_memory_used_bytes) / sum by (class) (llmgw_gpu_memory_total_bytes)`,
|
||||
"class", window, step); err == nil {
|
||||
f.GPUMemory = LineChart(s, "%", 6)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// modelEfficiency is what a model gets out of the GPUs it holds, over the
|
||||
// last quarter hour.
|
||||
type modelEfficiency struct {
|
||||
Model string
|
||||
// TokensPerSec is output tokens a second; GPUs the cards it holds now.
|
||||
TokensPerSec float64
|
||||
GPUs float64
|
||||
// Watts is what those cards draw; Util their average utilisation.
|
||||
Watts, Util float64
|
||||
// PerGPU and PerWh are the two ways of saying how much work the cards
|
||||
// do: output tokens a second per card, and per watt-hour drawn.
|
||||
PerGPU, PerWh float64
|
||||
}
|
||||
|
||||
// The queries behind the table. Each model's cards are found by joining the
|
||||
// GPU series to llmgw_gpu_assigned on the card, which carries the model.
|
||||
var efficiencyQueries = []struct {
|
||||
expr string
|
||||
into func(*modelEfficiency, float64)
|
||||
}{
|
||||
{`sum by (model) (rate(llmgw_tokens_total{direction="completion"}[15m]))`,
|
||||
func(e *modelEfficiency, v float64) { e.TokensPerSec = v }},
|
||||
{`count by (model) (llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.GPUs = v }},
|
||||
{`sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.Watts = v }},
|
||||
{`avg by (model) (llmgw_gpu_utilization_percent * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.Util = v }},
|
||||
}
|
||||
|
||||
// loadEfficiency is the table, best per card first; nil when the metrics
|
||||
// store cannot be read or no model holds a GPU.
|
||||
func loadEfficiency(ctx context.Context, h History) []modelEfficiency {
|
||||
if h == nil {
|
||||
return nil
|
||||
}
|
||||
byModel := map[string]*modelEfficiency{}
|
||||
for _, q := range efficiencyQueries {
|
||||
// The last of a short range: what History offers, and the one point
|
||||
// the table needs.
|
||||
series, err := h.RangeQuery(ctx, q.expr, "model", 10*time.Minute, 5*time.Minute)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
for _, s := range series {
|
||||
if len(s.Points) == 0 {
|
||||
continue
|
||||
}
|
||||
e := byModel[s.Name]
|
||||
if e == nil {
|
||||
e = &modelEfficiency{Model: s.Name}
|
||||
byModel[s.Name] = e
|
||||
}
|
||||
q.into(e, s.Last())
|
||||
}
|
||||
}
|
||||
var out []modelEfficiency
|
||||
for _, e := range byModel {
|
||||
if e.GPUs == 0 {
|
||||
continue // a model on the CPU; this table is about cards
|
||||
}
|
||||
e.PerGPU = e.TokensPerSec / e.GPUs
|
||||
if e.Watts > 0 {
|
||||
e.PerWh = e.TokensPerSec * 3600 / e.Watts
|
||||
}
|
||||
out = append(out, *e)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].PerGPU != out[j].PerGPU {
|
||||
return out[i].PerGPU > out[j].PerGPU
|
||||
}
|
||||
return out[i].Model < out[j].Model
|
||||
})
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"html/template"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// The GPU map is every card in the fleet, a square each, a row per node,
|
||||
// coloured by one figure: how busy, how hot, or how full its memory is. A
|
||||
// hundred cards in a table are a hundred rows to read; on the map the one
|
||||
// that is not like the others is seen before anything is read.
|
||||
|
||||
// gpuMapModes are what the map can be coloured by.
|
||||
var gpuMapModes = []struct{ Key, Label string }{
|
||||
{"util", "busy"}, {"memory", "memory"}, {"temp", "temperature"},
|
||||
}
|
||||
|
||||
type gpuMap struct {
|
||||
By string
|
||||
Modes []struct{ Key, Label string }
|
||||
Rows []gpuMapRow
|
||||
// Legend says what the colours mean in this mode.
|
||||
Legend string
|
||||
}
|
||||
|
||||
type gpuMapRow struct {
|
||||
Node string
|
||||
Cells []gpuCell
|
||||
}
|
||||
|
||||
type gpuCell struct {
|
||||
Index int
|
||||
// Model and Replica hold the card; empty is a free card.
|
||||
Model, Replica string
|
||||
// Style is the cell's colour, built here from numbers alone, which is
|
||||
// what makes it safe to put in a style attribute; Title says everything
|
||||
// known about the card.
|
||||
Style template.CSS
|
||||
Title string
|
||||
// Label is the figure written in the cell.
|
||||
Label string
|
||||
// Faulty is a card whose wear needs seeing to.
|
||||
Faulty bool
|
||||
}
|
||||
|
||||
// idleHolding is a replica holding GPUs that are doing nothing.
|
||||
type idleHolding struct {
|
||||
Model, Replica, Node string
|
||||
GPUs []int
|
||||
// MaxUtil is the busiest of its cards, now.
|
||||
MaxUtil float64
|
||||
// Idle is how long since a request was routed to it, or "" when not
|
||||
// known.
|
||||
Idle string
|
||||
}
|
||||
|
||||
// buildGPUMap draws the map from what the nodes last reported and where
|
||||
// replicas are. Nil for a fleet without GPUs.
|
||||
func buildGPUMap(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, by, prefix string) *gpuMap {
|
||||
held := heldBy(reps)
|
||||
m := &gpuMap{By: by, Modes: gpuMapModes}
|
||||
if m.By == "" {
|
||||
m.By = "util"
|
||||
}
|
||||
switch m.By {
|
||||
case "temp":
|
||||
m.Legend = "Blue is cool, red is 90 °C and over."
|
||||
case "memory":
|
||||
m.Legend = "Pale is empty, dark is full. Engines reserve most of a card's memory when they start."
|
||||
default:
|
||||
m.By = "util"
|
||||
m.Legend = "Pale is idle, dark green is fully busy. A dashed border is a card no replica holds."
|
||||
}
|
||||
for _, n := range nodes {
|
||||
if n.GPUTotal == 0 {
|
||||
continue
|
||||
}
|
||||
row := gpuMapRow{Node: strings.TrimPrefix(n.ID, prefix)}
|
||||
for i := range n.GPUTotal {
|
||||
c := gpuCell{Index: i}
|
||||
if h, ok := held[gpuKey(n.ID, i)]; ok {
|
||||
c.Model, c.Replica = h.model, h.replica
|
||||
}
|
||||
var title []string
|
||||
title = append(title, fmt.Sprintf("%s GPU %d", n.ID, i))
|
||||
if c.Model != "" {
|
||||
title = append(title, c.Model+" ("+c.Replica+")")
|
||||
} else {
|
||||
title = append(title, "free")
|
||||
}
|
||||
s, sampled := sensorOf(n, i)
|
||||
if sampled {
|
||||
if s.MemoryTotalMB > 0 && s.UtilPercent >= 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f%% busy", s.UtilPercent))
|
||||
}
|
||||
if s.MemoryTotalMB > 0 {
|
||||
title = append(title, fmt.Sprintf("%.1f of %.0f GB", s.MemoryUsedMB/1024, s.MemoryTotalMB/1024))
|
||||
}
|
||||
if s.TempC > 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f °C", s.TempC))
|
||||
}
|
||||
if s.PowerW > 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f W", s.PowerW))
|
||||
}
|
||||
if len(s.Throttle) > 0 {
|
||||
title = append(title, "held back by "+strings.Join(s.Throttle, ", "))
|
||||
}
|
||||
title = append(title, s.Problems...)
|
||||
}
|
||||
c.Title = strings.Join(title, " · ")
|
||||
style, label := cellColour(m.By, s, sampled)
|
||||
if c.Model == "" {
|
||||
style += ";border-style:dashed"
|
||||
}
|
||||
// A card wearing out is outlined whatever the map is coloured
|
||||
// by: it is the one thing on the map that needs doing.
|
||||
if len(s.Problems) > 0 {
|
||||
style += ";border:2px solid #c0392b"
|
||||
c.Faulty = true
|
||||
}
|
||||
c.Style, c.Label = template.CSS(style), label
|
||||
row.Cells = append(row.Cells, c)
|
||||
}
|
||||
m.Rows = append(m.Rows, row)
|
||||
}
|
||||
if len(m.Rows) == 0 {
|
||||
return nil
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// cellColour is a cell's background, and the figure written in it.
|
||||
func cellColour(by string, s sensorView, sampled bool) (string, string) {
|
||||
const unknown = "background:var(--panel-2,#eee)"
|
||||
switch by {
|
||||
case "temp":
|
||||
if !sampled || s.TempC <= 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
// 30 °C blue to 90 °C red.
|
||||
f := clamp01((s.TempC - 30) / 60)
|
||||
return fmt.Sprintf("background:hsl(%.0f,70%%,55%%)", 220-220*f), fmt.Sprintf("%.0f", s.TempC)
|
||||
case "memory":
|
||||
if !sampled || s.MemoryTotalMB <= 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
f := clamp01(s.MemoryUsedMB / s.MemoryTotalMB)
|
||||
return fmt.Sprintf("background:hsl(265,45%%,%.0f%%)", 90-55*f), fmt.Sprintf("%.0f", 100*f)
|
||||
}
|
||||
if !sampled || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
f := clamp01(s.UtilPercent / 100)
|
||||
return fmt.Sprintf("background:hsl(140,%.0f%%,%.0f%%)", 15+50*f, 88-50*f), fmt.Sprintf("%.0f", s.UtilPercent)
|
||||
}
|
||||
|
||||
// idleHoldings are the replicas whose GPUs all read under 5% busy now:
|
||||
// capacity held and not asked for, the first place to find room.
|
||||
func idleHoldings(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, prefix string) []idleHolding {
|
||||
byID := map[string]adminapi.NodeView{}
|
||||
for _, n := range nodes {
|
||||
byID[n.ID] = n
|
||||
}
|
||||
var out []idleHolding
|
||||
for _, r := range reps {
|
||||
if r.State != "ready" {
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
if len(sh.GPUs) == 0 {
|
||||
continue
|
||||
}
|
||||
h := idleHolding{Model: r.Model, Replica: r.ID, Node: strings.TrimPrefix(sh.Node, prefix), GPUs: sh.GPUs}
|
||||
idle, known := true, true
|
||||
for _, g := range sh.GPUs {
|
||||
s, ok := sensorOf(byID[sh.Node], g)
|
||||
if !ok || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
|
||||
known = false
|
||||
break
|
||||
}
|
||||
h.MaxUtil = max(h.MaxUtil, s.UtilPercent)
|
||||
if s.UtilPercent >= 5 {
|
||||
idle = false
|
||||
}
|
||||
}
|
||||
if !known || !idle {
|
||||
continue
|
||||
}
|
||||
if r.IdleSec >= 0 {
|
||||
h.Idle = (time.Duration(r.IdleSec) * time.Second).String()
|
||||
}
|
||||
out = append(out, h)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return len(out[i].GPUs) > len(out[j].GPUs) })
|
||||
return out
|
||||
}
|
||||
|
||||
type holder struct{ model, replica string }
|
||||
|
||||
func heldBy(reps []adminapi.ReplicaView) map[string]holder {
|
||||
out := map[string]holder{}
|
||||
for _, r := range reps {
|
||||
switch r.State {
|
||||
case "stopped", "failed":
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
for _, g := range sh.GPUs {
|
||||
out[gpuKey(sh.Node, g)] = holder{r.Model, r.ID}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func gpuKey(node string, gpu int) string { return fmt.Sprintf("%s/%d", node, gpu) }
|
||||
|
||||
// sensorView is a card's last sample.
|
||||
type sensorView = struct {
|
||||
TempC, PowerW, UtilPercent, MemoryUsedMB, MemoryTotalMB float64
|
||||
Throttle []string
|
||||
Problems []string
|
||||
}
|
||||
|
||||
func sensorOf(n adminapi.NodeView, gpu int) (sensorView, bool) {
|
||||
if n.Load == nil {
|
||||
return sensorView{}, false
|
||||
}
|
||||
for _, s := range n.Load.GPUs {
|
||||
if s.Index == gpu {
|
||||
return sensorView{TempC: s.TempC, PowerW: s.PowerW, UtilPercent: s.UtilPercent,
|
||||
MemoryUsedMB: s.MemoryUsedMB, MemoryTotalMB: s.MemoryTotalMB, Throttle: s.Throttle,
|
||||
Problems: s.Wear.Problems()}, true
|
||||
}
|
||||
}
|
||||
return sensorView{}, false
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
)
|
||||
|
||||
// mapFleet is a node of four cards: chat holds two and is idle, code holds
|
||||
// one and is busy, one is free.
|
||||
type mapFleet struct{ fakeSource }
|
||||
|
||||
func (mapFleet) Nodes() []adminapi.NodeView {
|
||||
busy := func(i int, util, temp float64) inventory.GPUSensor {
|
||||
return inventory.GPUSensor{Index: i, UtilPercent: util, TempC: temp, MemoryUsedMB: 40000, MemoryTotalMB: 81559, PowerW: 300}
|
||||
}
|
||||
return []adminapi.NodeView{{ID: "gpu07", State: "ready", GPUTotal: 4, GPUFree: 1, Classes: []string{"h100-80g"},
|
||||
Load: &adminapi.LoadView{GPUs: []inventory.GPUSensor{busy(0, 1, 35), busy(1, 0, 34), busy(2, 93, 81), worn(busy(3, 0, 30))}}}}
|
||||
}
|
||||
|
||||
func worn(s inventory.GPUSensor) inventory.GPUSensor {
|
||||
s.Wear = &inventory.GPUWear{RemapPending: true, XIDs: 1, LastXID: 79, LastXIDText: "GPU has fallen off the bus."}
|
||||
return s
|
||||
}
|
||||
|
||||
func (mapFleet) Replicas(string) []adminapi.ReplicaView {
|
||||
return []adminapi.ReplicaView{
|
||||
{ID: "chat-1", Model: "chat", State: "ready", IdleSec: 1800, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{0, 1}}}},
|
||||
{ID: "code-1", Model: "code", State: "ready", IdleSec: 2, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{2}}}},
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClusterPageMapsEveryCardAndNamesTheIdleOnes(t *testing.T) {
|
||||
_, body := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if n := strings.Count(body, `class="gpu-cell"`); n != 4 {
|
||||
t.Errorf("%d cards on the map, want 4", n)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"gpu07 GPU 2 · code (code-1) · 93% busy",
|
||||
"gpu07 GPU 3 · free",
|
||||
"memory rows remapped, waiting for a reset · XID 79 (GPU has fallen off the bus.)",
|
||||
"border:2px solid #c0392b",
|
||||
"border-style:dashed",
|
||||
"Cards held and doing nothing",
|
||||
"<code>chat-1</code></td><td><code>gpu07</code>", "30m0s ago",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "<code>code-1</code></td><td><code>gpu07</code>") {
|
||||
t.Error("a busy replica was listed as idle")
|
||||
}
|
||||
_, temp := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/?gpumap=temp", nil)
|
||||
if !strings.Contains(temp, "<strong>temperature</strong>") || !strings.Contains(temp, ">81</span>") {
|
||||
t.Error("the map coloured by temperature does not say so, or show it")
|
||||
}
|
||||
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(cpu, "gpu-map") {
|
||||
t.Error("a fleet without GPUs has a GPU map")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachModelSaysWhatItGetsFromItsCards(t *testing.T) {
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{}}
|
||||
for i, vals := range [][2]float64{{120, 30}, {2, 2}, {1200, 400}, {80, 40}} {
|
||||
h.byExpr[efficiencyQueries[i].expr] = []promq.Series{ts("chat", vals[0], vals[0]), ts("code", vals[1], vals[1])}
|
||||
}
|
||||
got := loadEfficiency(context.Background(), h)
|
||||
if len(got) != 2 || got[0].Model != "chat" {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
c := got[0]
|
||||
if c.PerGPU != 60 || c.PerWh != 120*3600/1200.0 || c.Util != 80 {
|
||||
t.Errorf("chat: %+v", c)
|
||||
}
|
||||
if got[1].PerGPU != 15 {
|
||||
t.Errorf("code: %+v", got[1])
|
||||
}
|
||||
}
|
||||
@@ -40,8 +40,10 @@ type nodeEngine struct {
|
||||
// nodeCharts are the node's recent history, from the metrics store.
|
||||
type nodeCharts struct {
|
||||
CPU, Memory, Load, EngineCPU, EngineMemory, Queue TimeSeries
|
||||
Window string
|
||||
Err string
|
||||
// The GPUs', one line per card; empty on a node without them.
|
||||
GPUUtil, GPUMemory, GPUTemp, GPUPower TimeSeries
|
||||
Window string
|
||||
Err string
|
||||
}
|
||||
|
||||
// nodePanels are the charts, each a query scoped to this node. %s is the
|
||||
@@ -63,6 +65,23 @@ var nodePanels = []struct {
|
||||
func(c *nodeCharts, t TimeSeries) { c.EngineMemory = t }},
|
||||
}
|
||||
|
||||
// gpuPanels are the GPUs' charts, one line per card, for a node that has
|
||||
// them. Read from the heartbeat's samples, not from what the node said when
|
||||
// it registered.
|
||||
var gpuPanels = []struct {
|
||||
expr, unit string
|
||||
into func(*nodeCharts, TimeSeries)
|
||||
}{
|
||||
{`llmgw_gpu_utilization_percent{node=%s} / 100`, "%",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUUtil = t }},
|
||||
{`llmgw_gpu_memory_used_bytes{node=%s}`, "bytes",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUMemory = t }},
|
||||
{`llmgw_gpu_temperature_celsius{node=%s}`, "",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUTemp = t }},
|
||||
{`llmgw_gpu_power_watts{node=%s}`, "",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUPower = t }},
|
||||
}
|
||||
|
||||
// NodePage shows one node. Its charts come from the metrics store when one
|
||||
// is configured; the current figures come from the node's last heartbeat
|
||||
// either way.
|
||||
@@ -115,14 +134,14 @@ func NodePage(src Source, history History, window time.Duration, onAdminListener
|
||||
sort.Slice(d.Engines, func(i, j int) bool { return d.Engines[i].CPUCores > d.Engines[j].CPUCores })
|
||||
|
||||
if history != nil {
|
||||
d.Charts = loadNodeCharts(r.Context(), history, id, replicas, window, log)
|
||||
d.Charts = loadNodeCharts(r.Context(), history, id, replicas, d.Node.GPUTotal, window, log)
|
||||
}
|
||||
noStore(w)
|
||||
render(w, "node", d)
|
||||
}
|
||||
}
|
||||
|
||||
func loadNodeCharts(ctx context.Context, h History, node string, replicas []string,
|
||||
func loadNodeCharts(ctx context.Context, h History, node string, replicas []string, gpus int,
|
||||
window time.Duration, log *slog.Logger,
|
||||
) *nodeCharts {
|
||||
step := window / 60
|
||||
@@ -139,6 +158,19 @@ func loadNodeCharts(ctx context.Context, h History, node string, replicas []stri
|
||||
}
|
||||
p.into(out, LineChart(series, p.unit, 4))
|
||||
}
|
||||
// Every card its own line, up to the eight a node commonly has.
|
||||
if gpus > 0 {
|
||||
for _, p := range gpuPanels {
|
||||
series, err := h.RangeQuery(ctx, fmt.Sprintf(p.expr, quoted), "gpu", window, step)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for i := range series {
|
||||
series[i].Name = "GPU " + series[i].Name
|
||||
}
|
||||
p.into(out, LineChart(series, p.unit, max(gpus, 8)))
|
||||
}
|
||||
}
|
||||
// The engines' queues, for the replicas that are here now.
|
||||
if len(replicas) > 0 {
|
||||
re := make([]string, len(replicas))
|
||||
|
||||
@@ -205,3 +205,68 @@ func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// gpuSource is a node with two GPUs, one busy and one idle.
|
||||
type gpuSource struct{ fakeSource }
|
||||
|
||||
func (gpuSource) Nodes() []adminapi.NodeView {
|
||||
n := fakeSource{}.Nodes()[0]
|
||||
n.GPUTotal = 2
|
||||
n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
|
||||
GPUs: []inventory.GPUSensor{
|
||||
{Index: 0, TempC: 71, PowerW: 612, PowerLimitW: 700, UtilPercent: 97, MemUtilPercent: 64,
|
||||
MemoryUsedMB: 72909, MemoryTotalMB: 81559, SMClockMHz: 1980, ECCUncorrected: 0},
|
||||
{Index: 1, TempC: 30, PowerW: 70, PowerLimitW: 700, UtilPercent: 0, MemUtilPercent: 0,
|
||||
MemoryUsedMB: 512, MemoryTotalMB: 81559, SMClockMHz: 345, ECCUncorrected: 3},
|
||||
}}
|
||||
return []adminapi.NodeView{n}
|
||||
}
|
||||
|
||||
func TestAGPUNodesPageShowsHowBusyEachCardIsAndOverTime(t *testing.T) {
|
||||
q := func(expr string) string { return strings.Replace(expr, "%s", `"llmgw-node01"`, 1) }
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{
|
||||
q(`llmgw_gpu_utilization_percent{node=%s} / 100`): {ts("0", 0.9, 0.97), ts("1", 0, 0)},
|
||||
q(`llmgw_gpu_memory_used_bytes{node=%s}`): {ts("0", 7e10, 7.6e10)},
|
||||
q(`llmgw_gpu_temperature_celsius{node=%s}`): {ts("0", 65, 71)},
|
||||
q(`llmgw_gpu_power_watts{node=%s}`): {ts("0", 500, 612)},
|
||||
}}
|
||||
_, body := get(t, nodeMux(gpuSource{}, h), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, want := range []string{
|
||||
"97%", "memory 64%", "71.2 <span class=\"note\">of 80 GB", "1980 MHz",
|
||||
"3 uncorrected ECC errors",
|
||||
"GPU busy, by card", "GPU memory used, by card", "GPU temperature, °C", "GPU power, W", "GPU 0",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// A node without GPUs is not asked about them.
|
||||
h2 := &fakeHistory{}
|
||||
get(t, nodeMux(cpuFleet{}, h2), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, e := range h2.asked {
|
||||
if strings.Contains(e, "llmgw_gpu_") {
|
||||
t.Errorf("a node with no GPU was asked for %s", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClusterPageDrawsItsGPUsByClass(t *testing.T) {
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{
|
||||
`avg by (class) (llmgw_gpu_utilization_percent) / 100`: {ts("h100-80g", 0.2, 0.6)},
|
||||
`sum by (class) (llmgw_gpu_memory_used_bytes) / sum by (class) (llmgw_gpu_memory_total_bytes)`: {ts("h100-80g", 0.8, 0.9)},
|
||||
}}
|
||||
_, body := get(t, Dashboard(fakeSource{}, nil, "v1", time.Second, true, NavExtras{History: h, Window: time.Hour}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if !strings.Contains(body, "GPUs busy, by class") || !strings.Contains(body, "GPU memory in use, by class") {
|
||||
t.Error("a fleet with GPUs has no GPU charts")
|
||||
}
|
||||
h2 := &fakeHistory{}
|
||||
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{History: h2, Window: time.Hour}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(cpu, "GPUs busy") {
|
||||
t.Error("a fleet without GPUs was drawn GPU charts")
|
||||
}
|
||||
for _, e := range h2.asked {
|
||||
if strings.Contains(e, "llmgw_gpu_") {
|
||||
t.Errorf("a fleet without GPUs was asked for %s", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// signupAccounts is an account service whose sign-up answers with err.
|
||||
type signupAccounts struct {
|
||||
fakeAccounts
|
||||
err error
|
||||
}
|
||||
|
||||
func (s signupAccounts) SignUp(context.Context, string, string, string) error { return s.err }
|
||||
func (signupAccounts) SignupOpen() bool { return true }
|
||||
|
||||
func signupBody(t *testing.T, err error) string {
|
||||
t.Helper()
|
||||
mux := http.NewServeMux()
|
||||
NewAccountPages(signupAccounts{err: err}, nil, nil, "v0", slog.New(slog.DiscardHandler)).Register(mux)
|
||||
form := url.Values{"email": {"a@b.c"}, "name": {"A"}, "password": {"long enough password"}}
|
||||
req := httptest.NewRequest(http.MethodPost, "/signup", strings.NewReader(form.Encode()))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec.Body.String()
|
||||
}
|
||||
|
||||
func TestASignupThatFailsSaysSoWithAReferenceAndNoDatabaseWords(t *testing.T) {
|
||||
body := signupBody(t, errors.New(`duplicate key value violates unique constraint "users_email_key"`))
|
||||
if strings.Contains(body, "users_email_key") || strings.Contains(body, "constraint") {
|
||||
t.Errorf("the database's own words reached the page: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "was not created") || !strings.Contains(body, "reference: err-") {
|
||||
t.Errorf("no plain failure with a reference to give an operator: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASignupSaysWhatToDoIfTheAddressHadAnAccount(t *testing.T) {
|
||||
// The service answers a taken address as it does a new one, so nobody
|
||||
// can find out from this page who has an account; what the page can do
|
||||
// is say what to do in either case.
|
||||
body := signupBody(t, nil)
|
||||
if !strings.Contains(body, "already had an account") || !strings.Contains(body, `href="/login"`) ||
|
||||
!strings.Contains(body, "reset it") {
|
||||
t.Errorf("the done page leaves someone who already had an account stuck: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -862,3 +862,10 @@ table.rows td.warn { color: var(--warn); cursor: help; }
|
||||
.held-card .adapters h4 { font-size: .9rem; margin: 0 0 .35rem; display: flex; gap: .75rem; align-items: baseline; }
|
||||
.held-card .adapter-form .field-row { align-items: flex-end; flex-wrap: wrap; gap: .5rem 1rem; }
|
||||
.held-card .adapter-form select { display: block; margin-top: .25rem; max-width: 28rem; }
|
||||
|
||||
/* The GPU map: a square per card, a row per node. */
|
||||
.gpu-map { display: flex; flex-direction: column; gap: 3px; margin: 6px 0; overflow-x: auto; }
|
||||
.gpu-row { display: flex; align-items: center; gap: 3px; }
|
||||
.gpu-node { width: 7em; flex: none; font-size: 12px; color: var(--muted, #666); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.gpu-cell { width: 26px; height: 26px; flex: none; border: 1px solid rgba(0,0,0,.18); border-radius: 4px;
|
||||
font-size: 10px; line-height: 26px; text-align: center; color: rgba(0,0,0,.7); cursor: default; }
|
||||
@@ -11,6 +11,11 @@
|
||||
<p class="note">You will not get an email from this gateway — it does not send any.
|
||||
Tell whoever you asked for access that you have signed up.</p>
|
||||
<p><a href="/login">Sign in</a> once you have been approved.</p>
|
||||
{{/* The same page whether or not the address already had an account, so
|
||||
it says what to do in either case without saying which this was. */}}
|
||||
<p class="note">If this address already had an account, nothing new was made:
|
||||
<a href="/login">sign in</a> with it, or, if you have forgotten its password, ask
|
||||
an operator to reset it.</p>
|
||||
{{else}}
|
||||
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
|
||||
<form method="post" autocomplete="on">
|
||||
|
||||
@@ -131,7 +131,7 @@ reachable from this machine.</p>
|
||||
<p class="error">Cannot be served as it stands: {{.Refusal}}.</p>
|
||||
{{else}}
|
||||
{{if .Warning}}<p class="note error-row">{{.Warning}}.</p>{{end}}
|
||||
<form method="post" action="/admin/mirror/serve">
|
||||
<form method="post" action="/admin/mirror/serve" class="serve-form" data-engine="{{.Engine}}" data-free-gpus="{{.FreeGPUs}}">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="source" value="{{.Source}}">
|
||||
<input type="hidden" name="engine" value="{{.Engine}}">
|
||||
@@ -163,8 +163,23 @@ reachable from this machine.</p>
|
||||
<label>Engine context pool <span class="opt">divided across slots</span>
|
||||
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>
|
||||
</div>
|
||||
{{/* Where it runs is a choice with consequences, so it is a choice: the
|
||||
GPU classes the fleet has, and the CPU for llama.cpp. A profile name
|
||||
field stood here once, and reading as though naming a profile "gpu"
|
||||
would put it on one, it put GGUF models on the CPU of GPU clusters. */}}
|
||||
<div class="field-row">
|
||||
<label>Runs on
|
||||
<select name="placement">
|
||||
{{range .Placements}}<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>{{end}}
|
||||
</select></label>
|
||||
<label>GPUs per replica <span class="opt">on one node</span>
|
||||
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>
|
||||
{{if eq .Engine "llamacpp"}}<label>Parallel requests <span class="opt">per replica</span>
|
||||
<input type="text" name="parallel" value="{{.Parallel}}" size="3"></label>{{end}}
|
||||
</div>
|
||||
{{if .FreeGPUs}}<label class="cpu-ok"><input type="checkbox" name="cpu_ok" value="yes">
|
||||
Run on the CPU anyway: {{.FreeGPUs}} GPUs are free and would sit unused</label>{{end}}
|
||||
<div class="field-row">
|
||||
<label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
|
||||
<label>Needs RAM (GB) <span class="opt">for the context pool beside it</span>
|
||||
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}" required></label>
|
||||
<label>CPU cores <span class="opt">reserved; the engine's threads</span>
|
||||
@@ -173,8 +188,44 @@ reachable from this machine.</p>
|
||||
<input type="text" name="selector_key" placeholder="role" size="8">
|
||||
<input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>
|
||||
</div>
|
||||
<p class="sub serve-summary" hidden></p>
|
||||
<button type="submit">Add to the catalog</button>
|
||||
</form>
|
||||
<script>
|
||||
// What it will run as, in one line, kept up to date as the form changes:
|
||||
// engine, where, how many cards, memory, context and how it is shared.
|
||||
(function () {
|
||||
const f = document.currentScript.previousElementSibling;
|
||||
if (!f || !f.classList.contains("serve-form")) return;
|
||||
const out = f.querySelector(".serve-summary"), ok = f.querySelector(".cpu-ok");
|
||||
const v = (n) => (f.elements[n] ? f.elements[n].value.trim() : "");
|
||||
const engine = { llamacpp: "llama.cpp", vllm: "vLLM", sglang: "SGLang" }[f.dataset.engine] || f.dataset.engine;
|
||||
function draw() {
|
||||
const place = v("placement"), gpus = parseInt(v("gpus"), 10) || 1;
|
||||
const onCPU = place === "cpu" || place === "";
|
||||
let where = "on the CPU";
|
||||
if (!onCPU) {
|
||||
const cls = place.replace(/^gpu:/, "");
|
||||
where = gpus > 1 ? "on " + gpus + " " + cls + " GPUs of one node" : "on 1 GPU " + cls;
|
||||
}
|
||||
const parts = [engine + " " + where];
|
||||
if (v("min_memory_gb")) parts.push(v("min_memory_gb") + " GB of host memory");
|
||||
if (onCPU && v("cpu_cores")) parts.push(v("cpu_cores") + " cores");
|
||||
const pool = parseInt(v("ctx_size"), 10), par = parseInt(v("parallel"), 10);
|
||||
if (pool && par) parts.push("context pool " + pool + " over " + par + " parallel requests (" + Math.floor(pool / par) + " each)");
|
||||
else if (pool) parts.push("context pool " + pool);
|
||||
out.textContent = "Will run as: " + parts.join(", ") + ".";
|
||||
const warn = onCPU && +f.dataset.freeGpus > 0;
|
||||
out.classList.toggle("error", warn);
|
||||
if (warn) out.textContent += " " + f.dataset.freeGpus + " GPUs are free; on the CPU this runs many times slower.";
|
||||
if (ok) ok.hidden = !onCPU;
|
||||
out.hidden = false;
|
||||
}
|
||||
f.addEventListener("input", draw);
|
||||
f.addEventListener("change", draw);
|
||||
draw();
|
||||
})();
|
||||
</script>
|
||||
<p class="note">The context pool is what the engine is given; llama.cpp divides
|
||||
it across its parallel slots, so the number one request may use is the smaller
|
||||
one above.</p>
|
||||
@@ -416,6 +467,7 @@ version and the old weights stay where they are.</p>
|
||||
{{range .RepoSets}}<tr>
|
||||
<td><code>{{.Name}}</code>
|
||||
{{if .Shards}}<span class="opt">{{.Shards}} shards, fetched together</span>{{end}}
|
||||
{{with .Companions}}<span class="opt" title="{{join .}}">with {{len .}} files the engine reads beside them</span>{{end}}
|
||||
{{if eq .Kind "adapter"}}<span class="tag new" title="a LoRA adapter: it runs on a {{.Arch}} model it was made for, not on its own">LoRA adapter · {{.Arch}}</span>
|
||||
{{else if eq .Kind "model"}}<span class="opt">model{{with .Arch}} · {{.}}{{end}}</span>{{end}}</td>
|
||||
<td class="num">{{bytes .Bytes}}</td>
|
||||
@@ -441,7 +493,8 @@ version and the old weights stay where they are.</p>
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<p class="note">Only the weights are listed; a repository is mostly a README and
|
||||
a tokenizer. Shards of one file are one row and one button — a fifteen-shard
|
||||
a tokenizer. A safetensors set is fetched with the files vLLM reads beside it —
|
||||
config.json, the tokenizer, the shard index — and never with code. Shards of one file are one row and one button — a fifteen-shard
|
||||
model is one decision, not fifteen. Two separate files are two rows because they
|
||||
are usually alternatives: one quantisation or the other, not both.</p>
|
||||
<p class="note">The source URLs are pinned to
|
||||
|
||||
@@ -63,6 +63,21 @@
|
||||
<p class="empty">Nothing recorded in this window.</p>
|
||||
{{end}}
|
||||
|
||||
{{with .GPUHours}}
|
||||
<h2>GPU time, by owner</h2>
|
||||
{{/* What each owner's models held, whether or not anybody asked them
|
||||
anything: a model kept loaded holds its cards all the same, and that is
|
||||
the time a GPU quota is about. */}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" $.GPUHeld "title" "GPUs held, by owner" "note" "The cards held by the replicas of each owner's models: models whose owner is a part of the quota tree."}}
|
||||
</div>
|
||||
<table class="rows">
|
||||
<tr><th>Owner</th><th class="num">GPU-hours</th><th class="num">Held now</th></tr>
|
||||
{{range .}}<tr><td>{{.Owner}}</td><td class="num">{{printf "%.1f" .Hours}}</td><td class="num">{{printf "%.0f" .Now}}</td></tr>{{end}}
|
||||
</table>
|
||||
<p class="note">From the metrics store, which keeps its history for as long as it is set to (fifteen days by default); a model with no owner is counted under “no owner”.</p>
|
||||
{{end}}
|
||||
|
||||
{{/* The charts fold their tail; the table does not. Both are here because they
|
||||
answer different questions, and the one that is easy to get wrong is
|
||||
thinking the four coloured lines are all of it. */}}
|
||||
|
||||
@@ -49,9 +49,40 @@
|
||||
</div>
|
||||
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
|
||||
|
||||
{{with .GPUMap}}
|
||||
<h2>GPUs <span class="opt">by
|
||||
{{range .Modes}}{{if eq .Key $.GPUMap.By}}<strong>{{.Label}}</strong>{{else}}<a href="?gpumap={{.Key}}">{{.Label}}</a>{{end}} {{end}}</span></h2>
|
||||
<div class="gpu-map">
|
||||
{{range .Rows}}<div class="gpu-row"><span class="gpu-node">{{.Node}}</span>{{range .Cells}}<span class="gpu-cell" style="{{.Style}}" title="{{.Title}}">{{.Label}}</span>{{end}}</div>
|
||||
{{end}}</div>
|
||||
<p class="note">{{.Legend}} A red outline is a card wearing out — XID errors, memory rows remapped or waiting for a reset, NVLink errors. The figure is the one coloured by; the rest of a card is under the pointer.</p>
|
||||
{{end}}
|
||||
{{with .Efficiency}}
|
||||
<h3>What each model gets from its cards <span class="opt">last 15 minutes</span></h3>
|
||||
{{/* Per card, because a model on eight cards making twice the tokens of one
|
||||
on two is the worse of the two; per watt-hour, because that is what the
|
||||
room is paying. */}}
|
||||
<table class="rows"><tr><th>Model</th><th class="num">GPUs</th><th class="num">Output tokens/s</th><th class="num">Per GPU</th><th class="num">Busy</th><th class="num">Draw</th><th class="num">Tokens per Wh</th></tr>
|
||||
{{range .}}<tr><td>{{.Model}}</td><td class="num">{{printf "%.0f" .GPUs}}</td><td class="num">{{printf "%.1f" .TokensPerSec}}</td>
|
||||
<td class="num">{{printf "%.1f" .PerGPU}}</td><td class="num">{{printf "%.0f" .Util}}%</td>
|
||||
<td class="num">{{if .Watts}}{{printf "%.0f" .Watts}} W{{else}}—{{end}}</td><td class="num">{{if .PerWh}}{{printf "%.0f" .PerWh}}{{else}}—{{end}}</td></tr>
|
||||
{{end}}</table>
|
||||
{{end}}
|
||||
{{with .IdleGPUs}}
|
||||
<h3>Cards held and doing nothing</h3>
|
||||
{{/* The first place to find room: replicas whose every card reads under 5%
|
||||
busy now. A model kept warm on purpose is here too; whether it should
|
||||
be is the operator's call. */}}
|
||||
<table class="rows"><tr><th>Model</th><th>Replica</th><th>Node</th><th class="num">GPUs</th><th class="num">Busiest card</th><th>Last request</th></tr>
|
||||
{{range .}}<tr><td>{{.Model}}</td><td class="clip"><code>{{.Replica}}</code></td><td><code>{{.Node}}</code></td>
|
||||
<td class="num">{{len .GPUs}}</td><td class="num">{{printf "%.0f" .MaxUtil}}%</td><td>{{if .Idle}}{{.Idle}} ago{{else}}<span class="note">not known</span>{{end}}</td></tr>
|
||||
{{end}}</table>
|
||||
{{end}}
|
||||
|
||||
<h2>Nodes</h2>
|
||||
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
|
||||
{{if .Nodes}}
|
||||
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
|
||||
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyGPUs}}<th>vLLM build</th>{{end}}{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
|
||||
{{range .Nodes}}<tr>
|
||||
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
|
||||
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
|
||||
@@ -66,6 +97,23 @@
|
||||
was never asked about its disk must not read as a full one. */}}
|
||||
<td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
|
||||
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
|
||||
{{/* Which vLLM build the node installs: left to its hardware, or chosen.
|
||||
What is installed beside it, so a choice and its outcome are read
|
||||
together. Only a GPU node has one; changing it is for an
|
||||
administrator, on the page reached through a session. */}}
|
||||
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{if and $.Menu.CSRF .EngineBuilds}}
|
||||
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
|
||||
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
|
||||
<input type="hidden" name="engine" value="vllm">
|
||||
<select name="build" title="the vLLM build this node installs">
|
||||
{{range engineBuilds "vllm"}}<option value="{{.}}"{{if eq . (index $node.EngineBuilds "vllm")}} selected{{end}}>{{buildLabel .}}</option>{{end}}
|
||||
</select>
|
||||
<noscript><button type="submit">Save</button></noscript>
|
||||
</form>
|
||||
{{end}}
|
||||
{{else}}—{{end}}</td>{{end}}
|
||||
{{if $.AnyClasses}}<td>{{join .Classes}}</td>{{end}}
|
||||
{{if $.AnyFabrics}}<td>{{join .Fabrics}}</td>{{end}}
|
||||
<td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
|
||||
@@ -75,6 +123,13 @@
|
||||
{{range .Provision}}{{if and (ne .State "ready") (ne .State "skipped")}}<tr><td colspan="12" class="note{{if or (eq .State "failed") (eq .State "unsupported")}} bad{{end}}">{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{.State}}: {{.Detail}}</td></tr>{{end}}{{end}}
|
||||
{{range .Warnings}}<tr><td colspan="12" class="note bad">{{.}}</td></tr>{{end}}
|
||||
{{end}}</table>
|
||||
<script>
|
||||
// A build chosen is saved as it is chosen, with the page reloaded on the
|
||||
// answer: one select, no separate button to forget.
|
||||
document.querySelectorAll("form.build-choice select").forEach(function (s) {
|
||||
s.addEventListener("change", function () { s.form.submit(); });
|
||||
});
|
||||
</script>
|
||||
{{else}}<p class="empty">No agents have registered.</p>{{end}}
|
||||
|
||||
<h2>Models</h2>
|
||||
@@ -188,6 +243,14 @@ unused nodes, which is what decides whether such a replica can land.</p>
|
||||
{{template "tschart" dict "t" .Ready "title" "Replicas ready, by model"
|
||||
"note" "A model deliberately stopped has no line at all, which is why an on-demand catalogue looks quiet here."}}
|
||||
</div>
|
||||
{{if $.AnyGPUs}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .GPUBusy "title" "GPUs busy, by class"
|
||||
"note" "The average card of each class. Low here with replicas placed is capacity held and not asked for; each node's page has its cards one by one."}}
|
||||
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory in use, by class"
|
||||
"note" "Engines reserve most of a card's memory when they start, so this follows replicas placed more than requests served."}}
|
||||
</div>
|
||||
{{end}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica"
|
||||
"note" "Requests accepted and not yet started. The first thing to move when a replica is falling behind."}}
|
||||
|
||||
@@ -35,13 +35,18 @@
|
||||
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
|
||||
{{with .Load}}{{if .GPUs}}
|
||||
<h2>GPUs</h2>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by, and wear</th></tr>
|
||||
{{range .GPUs}}<tr>
|
||||
<td class="num">{{.Index}}</td>
|
||||
{{/* From the last sample; a dash where the driver or an older agent did
|
||||
not say, rather than a zero nobody measured. */}}
|
||||
<td class="load">{{if and .MemoryTotalMB (ge .UtilPercent 0.0)}}{{template "minibar" pctbar .UtilPercent}}<span class="load-v">{{printf "%.0f" .UtilPercent}}%{{if ge .MemUtilPercent 0.0}} <span class="note">memory {{printf "%.0f" .MemUtilPercent}}%</span>{{end}}</span>{{else}}—{{end}}</td>
|
||||
<td class="load">{{if .MemoryTotalMB}}{{template "minibar" pctbar (percent .MemoryUsedMB .MemoryTotalMB)}}<span class="load-v">{{printf "%.1f" (gib .MemoryUsedMB)}} <span class="note">of {{printf "%.0f" (gib .MemoryTotalMB)}} GB</span></span>{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
|
||||
<td class="num">{{if .SMClockMHz}}{{printf "%.0f" .SMClockMHz}} MHz{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{.PCIeReplays}} PCIe retries</span>{{end}}{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{end}}{{end}}
|
||||
{{end}}
|
||||
@@ -82,9 +87,9 @@ the adapter came up: a number that grows is the one to look at.</p>
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<table class="rows compact gpu-list">
|
||||
<tr><th>GPU</th><th>Name</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
|
||||
<tr><th>GPU</th><th>Name</th><th class="num">Compute</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
|
||||
{{range .Node.GPUs}}<tr>
|
||||
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.NUMA}}</td>
|
||||
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.ComputeCapability}}</td><td class="num">{{.NUMA}}</td>
|
||||
<td>{{if .NearestNIC}}<code>{{.NearestNIC}}</code> <span class="note">{{.NICPath}}</span>{{else}}—{{end}}</td>
|
||||
<td>{{if .Domain}}<span class="state-ready">joined</span>{{if and .Bandwidth (ne .Bandwidth "Full")}} <span class="bad">{{.Bandwidth}} bandwidth</span>{{end}}{{else if .FabricState}}<span class="bad">{{.FabricState}}</span>{{else}}<span class="note">—</span>{{end}}</td>
|
||||
</tr>{{end}}
|
||||
@@ -120,6 +125,16 @@ traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS
|
||||
{{template "tschart" dict "t" .EngineCPU "title" "Cores busy, by model" "note" "Each engine's process group, the way top counts it."}}
|
||||
{{template "tschart" dict "t" .EngineMemory "title" "Engine memory, by model" "note" "Resident memory, including the weights mapped from disk. The kernel can take those pages back, so this can exceed RAM in use."}}
|
||||
</div>
|
||||
{{if $.Node.GPUTotal}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .GPUUtil "title" "GPU busy, by card" "note" "Of each sample period, the share a kernel was running. A card held by a replica and idle here is one nobody is asking."}}
|
||||
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory used, by card" "note" "Framebuffer in use: the weights and the KV cache an engine reserved, and anything not llmgw's."}}
|
||||
</div>
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .GPUTemp "title" "GPU temperature, °C" "note" "Throttling usually starts in the high 80s; see what holds a card back in the table above."}}
|
||||
{{template "tschart" dict "t" .GPUPower "title" "GPU power, W" "note" "Draw against the card's enforced limit; at the limit is sw_power_cap."}}
|
||||
</div>
|
||||
{{end}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .Load "title" "Load average" "note" "Above the core count means work is waiting for a CPU."}}
|
||||
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica" "note" "Requests accepted and not yet started, on the replicas here now."}}
|
||||
|
||||
+22
-1
@@ -26,6 +26,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -109,6 +110,8 @@ var funcs = template.FuncMap{
|
||||
return p
|
||||
},
|
||||
"ago": ago,
|
||||
// gib is mebibytes as gibibytes, for a GPU's memory.
|
||||
"gib": func(mib float64) float64 { return mib / 1024 },
|
||||
// percent is part of whole, 0 to 100, for a bar.
|
||||
"percent": func(part, whole float64) float64 {
|
||||
if whole <= 0 {
|
||||
@@ -118,7 +121,25 @@ var funcs = template.FuncMap{
|
||||
},
|
||||
// shortNode is a node's name without what every node's name starts with.
|
||||
"shortNode": strings.TrimPrefix,
|
||||
"age": age,
|
||||
// engineBuilds are the builds an engine can be given on a node, and
|
||||
// buildLabel says one in words.
|
||||
"engineBuilds": control.EngineBuilds,
|
||||
"buildLabel": func(b string) string {
|
||||
switch b {
|
||||
case "auto":
|
||||
return "auto: what the hardware runs"
|
||||
case "cu13":
|
||||
return "CUDA 13"
|
||||
case "cu129":
|
||||
return "CUDA 12.9"
|
||||
case "cu12":
|
||||
return "CUDA 12"
|
||||
case "none":
|
||||
return "none"
|
||||
}
|
||||
return b
|
||||
},
|
||||
"age": age,
|
||||
// pctbar is a percentage as the minibar template takes it: whole, and
|
||||
// held to 0..100 so a measurement a little over does not draw past its box.
|
||||
"pctbar": func(v float64) struct{ P int } {
|
||||
|
||||
Reference in new issue
Block a user