Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
372ef64889 | ||
|
|
82a63cfc6e | ||
|
|
ff554c7062 | ||
|
|
b6a802c03b | ||
|
|
bd36798b15 | ||
|
|
dc483d8c68 | ||
|
|
43d7c4eac1 | ||
|
|
86e9a1cb83 |
No files matched your search
+104
-26
@@ -24,6 +24,57 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
From the field report in issue #2, after a deployment on GPU nodes.
|
||||
|
||||
### GPU metrics that move, and charts of them
|
||||
|
||||
A GPU's utilisation and memory in use were read once, when the agent
|
||||
started, and published from then on: a card busy since looked as idle as it
|
||||
was at registration, and one idle since as busy. They are now sampled with
|
||||
the temperatures and power every heartbeat (ten seconds at most), with the
|
||||
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
|
||||
An agent too old to send them still has its registration figures published,
|
||||
and a figure the driver does not give is left out rather than shown as zero.
|
||||
|
||||
- **New series**: `llmgw_gpu_memory_utilization_percent`,
|
||||
`llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
|
||||
`llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
|
||||
sample.
|
||||
- **The node page** shows each card's utilisation, memory, clock and ECC
|
||||
errors in its table, and charts of utilisation, memory, temperature and
|
||||
power, one line per card.
|
||||
- **The cluster page** charts the fleet's GPUs by class: average utilisation
|
||||
and memory in use.
|
||||
- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
|
||||
utilisation, memory, memory bandwidth, temperature, power against the
|
||||
limit, clock, and what holds cards back.
|
||||
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
|
||||
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
|
||||
- **The cluster page's GPU map**: every card a square, a row per node,
|
||||
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
|
||||
with what holds it and its figures under the pointer; free cards dashed.
|
||||
- **Cards held and doing nothing**: the replicas whose every card reads under
|
||||
5% busy, with when they were last asked — the first place to find room.
|
||||
- **What each model gets from its cards**, on the cluster page and in
|
||||
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
|
||||
draw and average utilisation, over the last quarter hour.
|
||||
- **What wears a card out**, read every minute: XID errors from the kernel's
|
||||
log (counted from when the agent started), memory rows remapped or pages
|
||||
retired and whether either waits for a reset, a card out of spare rows,
|
||||
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
|
||||
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
|
||||
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
|
||||
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
|
||||
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
|
||||
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
|
||||
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
|
||||
`GPUNVLinkErrorsClimbing`.
|
||||
- **GPU time by owner** on the usage page: the cards each owner's models
|
||||
held over the window, charted and summed into GPU-hours, whether or not the
|
||||
models were asked anything — the time a GPU quota is about. From the
|
||||
metrics store, as far back as it keeps history.
|
||||
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
|
||||
each card and whose quota it counts against, for joining the GPU series to
|
||||
models and tenants.
|
||||
|
||||
### A GPU placement no longer runs on the CPU unnoticed
|
||||
|
||||
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
|
||||
@@ -34,6 +85,11 @@ reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||||
own `ldd` could not say it: the backend is opened at run time.
|
||||
- **After the upgrade**, engines the agent finds already running are held to
|
||||
the same check when it adopts them: one serving a GPU placement on the CPU
|
||||
is stopped, and its replacement starts through the check. The installed
|
||||
CUDA build is checked again within ten minutes, and installed afresh if it
|
||||
sees no GPU.
|
||||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||||
included). One that does not is stopped and reported failed: "placed on GPU
|
||||
@@ -68,8 +124,10 @@ file of HTML.
|
||||
it and the manifest carries it; a node checks every file of a set against
|
||||
it before using the set, and fetches one that differs again.
|
||||
- **`llmgw mirror add` takes several URLs** into one version.
|
||||
- **A Hugging Face repository page is refused as a source**, saying to mirror
|
||||
it or to give the repository id for vLLM to fetch itself.
|
||||
- **A Hugging Face repository page given as a source** is read as the
|
||||
repository id for a model vLLM or SGLang runs, which the engine downloads
|
||||
itself (the node then needs to reach Hugging Face), and refused for one
|
||||
llama.cpp runs, which needs a file.
|
||||
|
||||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||||
such a model, as before.
|
||||
@@ -123,23 +181,45 @@ leaving the model with nothing. Now:
|
||||
stopping — is waited for, as before.
|
||||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||||
|
||||
### Every GPU node gets a vLLM it can run, chosen on the cluster page
|
||||
### Every GPU node gets engine builds it can run, by itself
|
||||
|
||||
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
|
||||
where the driver (580+) and the GPUs (Turing+) allow, and otherwise the same
|
||||
version's CUDA 12.9 build from vLLM's GitHub release, pinned to GitHub's
|
||||
digest, with PyTorch's CUDA 12.9 torch (driver 525+, Volta+). One version
|
||||
across the cluster, so the flags llmgw builds — checked against that version
|
||||
— hold on every node.
|
||||
Which build of an engine a node installs follows its hardware — the CPU
|
||||
architecture, the driver, and the oldest GPU on it — from what each project
|
||||
publishes and how it was compiled. Each engine has a list of builds, best
|
||||
first; the node takes the first its hardware meets, and the first of those
|
||||
that installs and is seen to compute on the GPU. A build the table is wrong
|
||||
about costs its download, and the next one is tried.
|
||||
|
||||
| | llama.cpp | vLLM |
|
||||
|---|---|---|
|
||||
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
|
||||
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
|
||||
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
|
||||
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
|
||||
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
|
||||
| No NVIDIA GPU | the CPU build | none |
|
||||
|
||||
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
|
||||
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
|
||||
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
|
||||
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
|
||||
starts at 7.5.
|
||||
|
||||
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
|
||||
column shows the version installed, and the installation's detail says why
|
||||
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
|
||||
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
|
||||
vLLM is not placed on a V100 by accident.
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation in the new environment. A build without kernels for the card
|
||||
imports and prints its version, and failed only on its first model; now it
|
||||
fails to install, saying so.
|
||||
- **The build is chosen per node** on the cluster page (*vLLM build*: auto,
|
||||
CUDA 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`. The choice is kept in the store and sent
|
||||
to the node at once.
|
||||
operation for vLLM, the devices llama.cpp lists for its GPU builds.
|
||||
- **The build can still be chosen per node** on the cluster page (auto, CUDA
|
||||
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
|
||||
at once.
|
||||
- **Whether an engine computes on its GPUs is read from every process
|
||||
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
|
||||
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
|
||||
`CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
### Engines per node, and nodes that do not run them all
|
||||
|
||||
@@ -154,13 +234,10 @@ and `--ignore-checks` passed over every other problem with it.
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none: a CUDA 12 vLLM on
|
||||
Volta beside a CUDA 13 one on newer cards.
|
||||
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
|
||||
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
|
||||
the threshold had been CUDA 13's for every version, so the doctor's advice
|
||||
to set an older version could not work. A refusal now says which versions
|
||||
would.
|
||||
or labels a different version of an engine, or none — for exceptions; the
|
||||
build each GPU generation needs is chosen without it (above).
|
||||
- **The requirements follow the version**, as the table above sets out; the
|
||||
threshold had been CUDA 13's for every version.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
@@ -194,9 +271,10 @@ placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||||
placed — the replica stayed unplaced or ran on one card — and is now refused
|
||||
when the model is added, saying to write `gpus_per_node` instead; so is
|
||||
`nodes` above one.
|
||||
placed — the replica stayed unplaced or ran on one card. It is now read as
|
||||
`gpus_per_node`, so a profile written that way is placed on that many cards
|
||||
after the upgrade, rather than dropped; only both given and disagreeing, or
|
||||
`nodes` above one, is refused.
|
||||
|
||||
## v0.10.6 — 2026-10-01
|
||||
|
||||
|
||||
@@ -1550,7 +1550,36 @@ type GPUSensor struct {
|
||||
PowerLimitW float64 `protobuf:"fixed64,5,opt,name=power_limit_w,json=powerLimitW,proto3" json:"power_limit_w,omitempty"`
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
|
||||
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
|
||||
// How busy the card is, sampled with the rest, rather than read once at
|
||||
// registration. -1 is not reported by the driver. memory_total_mb is set
|
||||
// by every agent that sends these, so zero there is an older agent.
|
||||
UtilizationPercent float64 `protobuf:"fixed64,7,opt,name=utilization_percent,json=utilizationPercent,proto3" json:"utilization_percent,omitempty"`
|
||||
MemoryUtilizationPercent float64 `protobuf:"fixed64,8,opt,name=memory_utilization_percent,json=memoryUtilizationPercent,proto3" json:"memory_utilization_percent,omitempty"`
|
||||
MemoryUsedMb float64 `protobuf:"fixed64,9,opt,name=memory_used_mb,json=memoryUsedMb,proto3" json:"memory_used_mb,omitempty"`
|
||||
MemoryTotalMb float64 `protobuf:"fixed64,10,opt,name=memory_total_mb,json=memoryTotalMb,proto3" json:"memory_total_mb,omitempty"`
|
||||
SmClockMhz float64 `protobuf:"fixed64,11,opt,name=sm_clock_mhz,json=smClockMhz,proto3" json:"sm_clock_mhz,omitempty"`
|
||||
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
|
||||
EccUncorrected float64 `protobuf:"fixed64,12,opt,name=ecc_uncorrected,json=eccUncorrected,proto3" json:"ecc_uncorrected,omitempty"`
|
||||
// What wears the card out, read every minute; has_wear says these are
|
||||
// present. Counts of -1 are not reported for this card.
|
||||
HasWear bool `protobuf:"varint,13,opt,name=has_wear,json=hasWear,proto3" json:"has_wear,omitempty"`
|
||||
PcieReplays int64 `protobuf:"varint,14,opt,name=pcie_replays,json=pcieReplays,proto3" json:"pcie_replays,omitempty"`
|
||||
RetiredPagesSbe int64 `protobuf:"varint,15,opt,name=retired_pages_sbe,json=retiredPagesSbe,proto3" json:"retired_pages_sbe,omitempty"`
|
||||
RetiredPagesDbe int64 `protobuf:"varint,16,opt,name=retired_pages_dbe,json=retiredPagesDbe,proto3" json:"retired_pages_dbe,omitempty"`
|
||||
RetirementPending bool `protobuf:"varint,17,opt,name=retirement_pending,json=retirementPending,proto3" json:"retirement_pending,omitempty"`
|
||||
RemappedRowsCorrectable int64 `protobuf:"varint,18,opt,name=remapped_rows_correctable,json=remappedRowsCorrectable,proto3" json:"remapped_rows_correctable,omitempty"`
|
||||
RemappedRowsUncorrectable int64 `protobuf:"varint,19,opt,name=remapped_rows_uncorrectable,json=remappedRowsUncorrectable,proto3" json:"remapped_rows_uncorrectable,omitempty"`
|
||||
RemapPending bool `protobuf:"varint,20,opt,name=remap_pending,json=remapPending,proto3" json:"remap_pending,omitempty"`
|
||||
RemapFailure bool `protobuf:"varint,21,opt,name=remap_failure,json=remapFailure,proto3" json:"remap_failure,omitempty"`
|
||||
NvlinkCrcErrors int64 `protobuf:"varint,22,opt,name=nvlink_crc_errors,json=nvlinkCrcErrors,proto3" json:"nvlink_crc_errors,omitempty"`
|
||||
NvlinkReplayErrors int64 `protobuf:"varint,23,opt,name=nvlink_replay_errors,json=nvlinkReplayErrors,proto3" json:"nvlink_replay_errors,omitempty"`
|
||||
NvlinkRecoveryErrors int64 `protobuf:"varint,24,opt,name=nvlink_recovery_errors,json=nvlinkRecoveryErrors,proto3" json:"nvlink_recovery_errors,omitempty"`
|
||||
// xid_errors are the driver's error events since the agent started.
|
||||
XidErrors int64 `protobuf:"varint,25,opt,name=xid_errors,json=xidErrors,proto3" json:"xid_errors,omitempty"`
|
||||
LastXid int32 `protobuf:"varint,26,opt,name=last_xid,json=lastXid,proto3" json:"last_xid,omitempty"`
|
||||
LastXidText string `protobuf:"bytes,27,opt,name=last_xid_text,json=lastXidText,proto3" json:"last_xid_text,omitempty"`
|
||||
LastXidUnix int64 `protobuf:"varint,28,opt,name=last_xid_unix,json=lastXidUnix,proto3" json:"last_xid_unix,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -1627,6 +1656,160 @@ func (x *GPUSensor) GetThrottle() []string {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetUtilizationPercent() float64 {
|
||||
if x != nil {
|
||||
return x.UtilizationPercent
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryUtilizationPercent() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryUtilizationPercent
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryUsedMb() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryUsedMb
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryTotalMb() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryTotalMb
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetSmClockMhz() float64 {
|
||||
if x != nil {
|
||||
return x.SmClockMhz
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetEccUncorrected() float64 {
|
||||
if x != nil {
|
||||
return x.EccUncorrected
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetHasWear() bool {
|
||||
if x != nil {
|
||||
return x.HasWear
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetPcieReplays() int64 {
|
||||
if x != nil {
|
||||
return x.PcieReplays
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetiredPagesSbe() int64 {
|
||||
if x != nil {
|
||||
return x.RetiredPagesSbe
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetiredPagesDbe() int64 {
|
||||
if x != nil {
|
||||
return x.RetiredPagesDbe
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetirementPending() bool {
|
||||
if x != nil {
|
||||
return x.RetirementPending
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemappedRowsCorrectable() int64 {
|
||||
if x != nil {
|
||||
return x.RemappedRowsCorrectable
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemappedRowsUncorrectable() int64 {
|
||||
if x != nil {
|
||||
return x.RemappedRowsUncorrectable
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemapPending() bool {
|
||||
if x != nil {
|
||||
return x.RemapPending
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemapFailure() bool {
|
||||
if x != nil {
|
||||
return x.RemapFailure
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkCrcErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkCrcErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkReplayErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkReplayErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkRecoveryErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkRecoveryErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetXidErrors() int64 {
|
||||
if x != nil {
|
||||
return x.XidErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXid() int32 {
|
||||
if x != nil {
|
||||
return x.LastXid
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXidText() string {
|
||||
if x != nil {
|
||||
return x.LastXidText
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXidUnix() int64 {
|
||||
if x != nil {
|
||||
return x.LastXidUnix
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
type ShardLoad struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -3999,14 +4182,39 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
" \x01(\x01R\tcpuPowerW\x12 \n" +
|
||||
"\finlet_temp_c\x18\v \x01(\x01R\n" +
|
||||
"inletTempC\x12'\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xb5\x01\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xde\b\n" +
|
||||
"\tGPUSensor\x12\x14\n" +
|
||||
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x15\n" +
|
||||
"\x06temp_c\x18\x02 \x01(\x01R\x05tempC\x12\"\n" +
|
||||
"\rmemory_temp_c\x18\x03 \x01(\x01R\vmemoryTempC\x12\x17\n" +
|
||||
"\apower_w\x18\x04 \x01(\x01R\x06powerW\x12\"\n" +
|
||||
"\rpower_limit_w\x18\x05 \x01(\x01R\vpowerLimitW\x12\x1a\n" +
|
||||
"\bthrottle\x18\x06 \x03(\tR\bthrottle\"m\n" +
|
||||
"\bthrottle\x18\x06 \x03(\tR\bthrottle\x12/\n" +
|
||||
"\x13utilization_percent\x18\a \x01(\x01R\x12utilizationPercent\x12<\n" +
|
||||
"\x1amemory_utilization_percent\x18\b \x01(\x01R\x18memoryUtilizationPercent\x12$\n" +
|
||||
"\x0ememory_used_mb\x18\t \x01(\x01R\fmemoryUsedMb\x12&\n" +
|
||||
"\x0fmemory_total_mb\x18\n" +
|
||||
" \x01(\x01R\rmemoryTotalMb\x12 \n" +
|
||||
"\fsm_clock_mhz\x18\v \x01(\x01R\n" +
|
||||
"smClockMhz\x12'\n" +
|
||||
"\x0fecc_uncorrected\x18\f \x01(\x01R\x0eeccUncorrected\x12\x19\n" +
|
||||
"\bhas_wear\x18\r \x01(\bR\ahasWear\x12!\n" +
|
||||
"\fpcie_replays\x18\x0e \x01(\x03R\vpcieReplays\x12*\n" +
|
||||
"\x11retired_pages_sbe\x18\x0f \x01(\x03R\x0fretiredPagesSbe\x12*\n" +
|
||||
"\x11retired_pages_dbe\x18\x10 \x01(\x03R\x0fretiredPagesDbe\x12-\n" +
|
||||
"\x12retirement_pending\x18\x11 \x01(\bR\x11retirementPending\x12:\n" +
|
||||
"\x19remapped_rows_correctable\x18\x12 \x01(\x03R\x17remappedRowsCorrectable\x12>\n" +
|
||||
"\x1bremapped_rows_uncorrectable\x18\x13 \x01(\x03R\x19remappedRowsUncorrectable\x12#\n" +
|
||||
"\rremap_pending\x18\x14 \x01(\bR\fremapPending\x12#\n" +
|
||||
"\rremap_failure\x18\x15 \x01(\bR\fremapFailure\x12*\n" +
|
||||
"\x11nvlink_crc_errors\x18\x16 \x01(\x03R\x0fnvlinkCrcErrors\x120\n" +
|
||||
"\x14nvlink_replay_errors\x18\x17 \x01(\x03R\x12nvlinkReplayErrors\x124\n" +
|
||||
"\x16nvlink_recovery_errors\x18\x18 \x01(\x03R\x14nvlinkRecoveryErrors\x12\x1d\n" +
|
||||
"\n" +
|
||||
"xid_errors\x18\x19 \x01(\x03R\txidErrors\x12\x19\n" +
|
||||
"\blast_xid\x18\x1a \x01(\x05R\alastXid\x12\"\n" +
|
||||
"\rlast_xid_text\x18\x1b \x01(\tR\vlastXidText\x12\"\n" +
|
||||
"\rlast_xid_unix\x18\x1c \x01(\x03R\vlastXidUnix\"m\n" +
|
||||
"\tShardLoad\x12\x19\n" +
|
||||
"\bshard_id\x18\x01 \x01(\tR\ashardId\x12\x1b\n" +
|
||||
"\tcpu_cores\x18\x02 \x01(\x01R\bcpuCores\x12(\n" +
|
||||
|
||||
@@ -304,6 +304,35 @@ message GPUSensor {
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
repeated string throttle = 6;
|
||||
// How busy the card is, sampled with the rest, rather than read once at
|
||||
// registration. -1 is not reported by the driver. memory_total_mb is set
|
||||
// by every agent that sends these, so zero there is an older agent.
|
||||
double utilization_percent = 7;
|
||||
double memory_utilization_percent = 8;
|
||||
double memory_used_mb = 9;
|
||||
double memory_total_mb = 10;
|
||||
double sm_clock_mhz = 11;
|
||||
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
|
||||
double ecc_uncorrected = 12;
|
||||
// What wears the card out, read every minute; has_wear says these are
|
||||
// present. Counts of -1 are not reported for this card.
|
||||
bool has_wear = 13;
|
||||
int64 pcie_replays = 14;
|
||||
int64 retired_pages_sbe = 15;
|
||||
int64 retired_pages_dbe = 16;
|
||||
bool retirement_pending = 17;
|
||||
int64 remapped_rows_correctable = 18;
|
||||
int64 remapped_rows_uncorrectable = 19;
|
||||
bool remap_pending = 20;
|
||||
bool remap_failure = 21;
|
||||
int64 nvlink_crc_errors = 22;
|
||||
int64 nvlink_replay_errors = 23;
|
||||
int64 nvlink_recovery_errors = 24;
|
||||
// xid_errors are the driver's error events since the agent started.
|
||||
int64 xid_errors = 25;
|
||||
int32 last_xid = 26;
|
||||
string last_xid_text = 27;
|
||||
int64 last_xid_unix = 28;
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
|
||||
@@ -487,6 +487,427 @@
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "GPUs",
|
||||
"id": 14,
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 30,
|
||||
"w": 24,
|
||||
"h": 1
|
||||
},
|
||||
"collapsed": false,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU busy, by card",
|
||||
"description": "Share of each sample period a kernel was running, sampled every heartbeat.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 31,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 15,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_utilization_percent",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU memory in use, by card",
|
||||
"description": "Framebuffer in use. Engines reserve most of it at start.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 31,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 16,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_memory_used_bytes / llmgw_gpu_memory_total_bytes",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU memory bandwidth busy, by card",
|
||||
"description": "Decode is bound by memory: high here with compute low is a card waiting on memory.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 38,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 17,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_memory_utilization_percent",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU temperature",
|
||||
"description": "Throttling usually starts in the high 80s.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 38,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 18,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_temperature_celsius",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "celsius"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU power against its limit",
|
||||
"description": "1 is the enforced limit; a card held there is power-capped.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 45,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 19,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1.1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU clock",
|
||||
"description": "A clock held low under load is the card throttled; see the next panel for why.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 45,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 20,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_sm_clock_mhz",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "rotmhz"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "What holds GPUs back",
|
||||
"description": "Cards throttled now, by reason, and uncorrected ECC errors since the driver loaded.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 52,
|
||||
"w": 24,
|
||||
"h": 7
|
||||
},
|
||||
"id": 21,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (reason) (llmgw_gpu_throttled)",
|
||||
"legendFormat": "throttled: {{reason}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum by (node, gpu) (llmgw_gpu_ecc_errors)",
|
||||
"legendFormat": "ECC errors {{node}} gpu {{gpu}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Output tokens/s per GPU, by model",
|
||||
"description": "What each model gets out of each card it holds.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 59,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 22,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) / count by (model) (llmgw_gpu_assigned)",
|
||||
"legendFormat": "{{model}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Output tokens per watt-hour, by model",
|
||||
"description": "Tokens for the energy the model's cards draw.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 59,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 23,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) * 3600 / sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)",
|
||||
"legendFormat": "{{model}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU wear",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"id": 24,
|
||||
"description": "What wears cards out: XID errors in the last 15 minutes, memory rows remapped, PCIe retries an hour. Anything above zero here is worth a look on that card's node page.",
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 66,
|
||||
"w": 24,
|
||||
"h": 7
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_xid_errors[15m]) > 0",
|
||||
"legendFormat": "XID {{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum by (node, gpu) (llmgw_gpu_remapped_rows) > 0",
|
||||
"legendFormat": "rows remapped {{node}} gpu {{gpu}}",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_pcie_replays[1h]) > 0",
|
||||
"legendFormat": "PCIe retries/h {{node}} gpu {{gpu}}",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_nvlink_errors[1h]) > 0",
|
||||
"legendFormat": "NVLink {{kind}}/h {{node}} gpu {{gpu}}",
|
||||
"refId": "D"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -200,6 +200,94 @@ groups:
|
||||
is skipped by placement rather than failing, so this is quiet until
|
||||
somebody wonders why a model will not land there.
|
||||
|
||||
- name: llmgw-gpus
|
||||
rules:
|
||||
- alert: GPUThermalThrottling
|
||||
expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})"
|
||||
description: >-
|
||||
Ten minutes held back by temperature: every replica on the card is
|
||||
slower than its measurements say. Usually airflow, a fan, or the
|
||||
room — the node page shows the card's temperature over time.
|
||||
|
||||
- alert: GPUUncorrectedECCErrors
|
||||
expr: llmgw_gpu_ecc_errors > 0
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors"
|
||||
description: >-
|
||||
Uncorrected memory errors since the driver loaded: what runs on the
|
||||
card may be computing on corrupted memory. Drain the node
|
||||
(llmgw ctl drain) and have the card looked at.
|
||||
|
||||
- alert: GPUXidError
|
||||
expr: increase(llmgw_gpu_xid_errors[15m]) > 0
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
|
||||
description: >-
|
||||
The driver reported an error on the card (its code is
|
||||
llmgw_gpu_last_xid, and the node page says it in words). 79 is a
|
||||
card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
|
||||
94/95 contained and uncontained ECC errors: those need the card
|
||||
looked at. Others are often an engine's own fault, and pass.
|
||||
|
||||
- alert: GPUOutOfSpareRows
|
||||
expr: llmgw_gpu_remap_failure == 1
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
|
||||
description: >-
|
||||
The card cannot remap another failing row. Drain the node and
|
||||
have the card replaced.
|
||||
|
||||
- alert: GPUNeedsReset
|
||||
expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
|
||||
description: >-
|
||||
Rows remapped or pages retired take effect only after the card is
|
||||
reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
|
||||
it back.
|
||||
|
||||
- alert: GPUPCIeRetriesClimbing
|
||||
expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
|
||||
description: >-
|
||||
The card's link to the host is retrying. A riser, the slot or a
|
||||
cable is the usual cause, and transfers to the card slow down.
|
||||
|
||||
- alert: GPUNVLinkErrorsClimbing
|
||||
expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
|
||||
description: >-
|
||||
Errors on the card's NVLink links: tensor-parallel replicas on it
|
||||
slow down, and a link that keeps failing takes a replica with it.
|
||||
|
||||
- alert: GPUHot
|
||||
expr: llmgw_gpu_temperature_celsius > 87
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C"
|
||||
description: >-
|
||||
Ten minutes above 87 °C, where most data-centre cards begin to
|
||||
throttle.
|
||||
|
||||
- name: monitoring-host
|
||||
rules:
|
||||
- alert: MonitoringDiskFilling
|
||||
|
||||
+18
-13
@@ -367,7 +367,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
|
||||
| Engine | On | How |
|
||||
|---|---|---|
|
||||
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
|
||||
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
|
||||
@@ -376,13 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
|
||||
run it — an older driver, a V100 — gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
|
||||
(7.0) or newer. Either way the install ends by computing something on the
|
||||
node's GPU, and a build that cannot fails to install, saying why;
|
||||
- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
|
||||
the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
|
||||
CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
|
||||
publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
|
||||
(7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
|
||||
**vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
|
||||
kernels — and the cluster page shows it running that version; run `llmgw
|
||||
ctl engine-check vllm` there once, since llmgw's flags are checked against
|
||||
the current one. Pascal and older run no vLLM. Whatever is chosen, the
|
||||
install ends by computing something on the GPU; a build that cannot is
|
||||
passed over for the next;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -427,8 +431,8 @@ cluster's for them, in order:
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- gpu_class: a10g # pin a class to a version you have tested
|
||||
engines: { vllm: "0.29.0" }
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
@@ -1269,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
|
||||
no build from 0.20 on has Volta kernels, and the node picks the newest that
|
||||
has. llmgw's flags are checked against the current version, so run `llmgw
|
||||
ctl engine-check vllm` on a V100 node once.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
+39
-2
@@ -104,6 +104,8 @@ type Agent struct {
|
||||
// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
|
||||
// from nvidia-smi.
|
||||
sensors *discovery.SensorSampler
|
||||
// wear reads what wears a GPU out: remapped rows, PCIe retries, XIDs.
|
||||
wear *discovery.HealthSampler
|
||||
|
||||
// fetchWatchers are the shards waiting on each source, so progress on one
|
||||
// download reaches every shard that asked for it. A fetch is single-flight
|
||||
@@ -150,6 +152,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
provKick: make(chan struct{}, 1),
|
||||
load: discovery.NewLoadSampler(),
|
||||
sensors: discovery.NewSensorSampler(),
|
||||
wear: discovery.NewHealthSampler(),
|
||||
}
|
||||
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
|
||||
return a, nil
|
||||
@@ -314,6 +317,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
return
|
||||
}
|
||||
// An engine started by an agent older than the GPU check may be one
|
||||
// serving a GPU placement on the CPU. Stopped, so its replacement starts
|
||||
// through the check and fails there in words, if the build still sees
|
||||
// no GPU — rather than going on unnoticed after the upgrade that could
|
||||
// have caught it.
|
||||
if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
|
||||
a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
_ = a.sup.Stop(ctx, o.ShardID)
|
||||
return
|
||||
}
|
||||
a.log.Info("adopted engine is serving",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID)
|
||||
}
|
||||
@@ -645,11 +659,34 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
|
||||
if a.sensors != nil {
|
||||
se := a.sensors.Sample(context.Background())
|
||||
out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
|
||||
var wear map[int]discovery.GPUWear
|
||||
if a.wear != nil && len(se.GPUs) > 0 {
|
||||
wear = map[int]discovery.GPUWear{}
|
||||
for _, w := range a.wear.Sample(context.Background()) {
|
||||
wear[w.Index] = w
|
||||
}
|
||||
}
|
||||
for _, g := range se.GPUs {
|
||||
out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
|
||||
gs := &llmgwv1.GPUSensor{
|
||||
Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
|
||||
PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
|
||||
})
|
||||
UtilizationPercent: g.UtilPercent, MemoryUtilizationPercent: g.MemUtilPercent,
|
||||
MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
|
||||
SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
|
||||
}
|
||||
if w, ok := wear[g.Index]; ok {
|
||||
gs.HasWear = true
|
||||
gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
|
||||
w.PCIeReplays, w.RetiredSBE, w.RetiredDBE, w.RetirePending
|
||||
gs.RemappedRowsCorrectable, gs.RemappedRowsUncorrectable, gs.RemapPending, gs.RemapFailure =
|
||||
w.RemappedCorrectable, w.RemappedUncorrectable, w.RemapPending, w.RemapFailure
|
||||
gs.NvlinkCrcErrors, gs.NvlinkReplayErrors, gs.NvlinkRecoveryErrors = w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery
|
||||
gs.XidErrors, gs.LastXid, gs.LastXidText = w.XIDs, int32(w.LastXID), w.LastXIDText
|
||||
if !w.LastXIDAt.IsZero() {
|
||||
gs.LastXidUnix = w.LastXIDAt.Unix()
|
||||
}
|
||||
}
|
||||
out.Gpus = append(out.Gpus, gs)
|
||||
}
|
||||
}
|
||||
return out
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// GPUWear is what wears a card out or takes it down, beyond temperature:
|
||||
// read every minute, from nvidia-smi and the kernel's log. The early signs of
|
||||
// a card failing are here — a growing count of rows its memory had to remap,
|
||||
// PCIe link retries, XID errors — long before a replica on it fails.
|
||||
//
|
||||
// A count of -1 is not reported: a card without row remapping (before
|
||||
// Ampere) has no such count, which is not the same as none.
|
||||
type GPUWear struct {
|
||||
Index int
|
||||
// PCIeReplays are link-level retries on the card's PCIe link: a few are
|
||||
// noise, a count that keeps climbing is a riser, slot or cable.
|
||||
PCIeReplays int64
|
||||
// RetiredSBE and RetiredDBE are memory pages retired for single and
|
||||
// double bit errors (cards before Ampere); RetirePending is a page
|
||||
// waiting for a reset to be retired.
|
||||
RetiredSBE, RetiredDBE int64
|
||||
RetirePending bool
|
||||
// RemappedCorrectable and RemappedUncorrectable are memory rows remapped
|
||||
// (Ampere and later). RemapPending waits for a reset; RemapFailure is a
|
||||
// card out of spare rows — it needs replacing.
|
||||
RemappedCorrectable, RemappedUncorrectable int64
|
||||
RemapPending, RemapFailure bool
|
||||
// NVLink errors summed over the card's links, by kind.
|
||||
NVLinkCRC, NVLinkReplay, NVLinkRecovery int64
|
||||
// XIDs are the driver's error events for this card since the agent
|
||||
// started, and the last of them.
|
||||
XIDs int64
|
||||
LastXID int
|
||||
LastXIDText string
|
||||
LastXIDAt time.Time
|
||||
}
|
||||
|
||||
// healthEvery is how often health is read: it changes over hours, and the
|
||||
// kernel log and nvidia-smi's full report are the heavier reads.
|
||||
const healthEvery = time.Minute
|
||||
|
||||
// HealthSampler reads GPUWear, keeping the XID counts between reads.
|
||||
type HealthSampler struct {
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
at time.Time
|
||||
last []GPUWear
|
||||
since time.Time
|
||||
xids map[int]*GPUWear
|
||||
started bool
|
||||
}
|
||||
|
||||
// NewHealthSampler reads the machine's own nvidia-smi and kernel log.
|
||||
func NewHealthSampler() *HealthSampler {
|
||||
return &HealthSampler{now: time.Now, xids: map[int]*GPUWear{},
|
||||
run: func(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
return exec.CommandContext(ctx, name, args...).Output() //nolint:gosec // fixed tools
|
||||
}}
|
||||
}
|
||||
|
||||
// Sample is the cards' health, read again once healthEvery has passed. Nil
|
||||
// on a machine without nvidia-smi.
|
||||
func (h *HealthSampler) Sample(ctx context.Context) []GPUWear {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
now := h.now()
|
||||
if !h.at.IsZero() && now.Sub(h.at) < healthEvery {
|
||||
return h.last
|
||||
}
|
||||
h.at = now
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
report, err := h.run(ctx, "nvidia-smi", "-q", "-x")
|
||||
if err != nil {
|
||||
h.last = nil
|
||||
return nil
|
||||
}
|
||||
cards, bus := parseHealthReport(report)
|
||||
if out, err := h.run(ctx, "nvidia-smi", "nvlink", "-e"); err == nil {
|
||||
for i, e := range parseNVLinkErrors(string(out)) {
|
||||
if i < len(cards) {
|
||||
cards[i].NVLinkCRC, cards[i].NVLinkReplay, cards[i].NVLinkRecovery = e[0], e[1], e[2]
|
||||
}
|
||||
}
|
||||
}
|
||||
h.readXIDs(ctx, now, bus)
|
||||
for i := range cards {
|
||||
if x := h.xids[i]; x != nil {
|
||||
cards[i].XIDs, cards[i].LastXID, cards[i].LastXIDText, cards[i].LastXIDAt = x.XIDs, x.LastXID, x.LastXIDText, x.LastXIDAt
|
||||
}
|
||||
}
|
||||
h.last = cards
|
||||
return cards
|
||||
}
|
||||
|
||||
// readXIDs adds the XID events logged since the last read. The first read
|
||||
// starts from now: what happened before the agent started was before, and
|
||||
// counting the whole log would raise every alert on every restart.
|
||||
func (h *HealthSampler) readXIDs(ctx context.Context, now time.Time, bus map[string]int) {
|
||||
if !h.started {
|
||||
h.started, h.since = true, now
|
||||
return
|
||||
}
|
||||
since := h.since
|
||||
h.since = now
|
||||
out, err := h.run(ctx, "journalctl", "-k", "--no-pager", "-o", "short-unix",
|
||||
"--since", "@"+strconv.FormatInt(since.Unix(), 10), "-g", "NVRM: Xid")
|
||||
if err != nil {
|
||||
// No journal, or nothing matched (journalctl exits 1 for that).
|
||||
return
|
||||
}
|
||||
for _, e := range parseXIDs(string(out)) {
|
||||
i, ok := bus[e.bus]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
x := h.xids[i]
|
||||
if x == nil {
|
||||
x = &GPUWear{Index: i}
|
||||
h.xids[i] = x
|
||||
}
|
||||
x.XIDs++
|
||||
x.LastXID, x.LastXIDText = e.code, e.text
|
||||
x.LastXIDAt = now
|
||||
if e.at > 0 {
|
||||
x.LastXIDAt = time.Unix(int64(e.at), 0)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// healthLog is the part of nvidia-smi's XML report health is read from.
|
||||
type healthLog struct {
|
||||
GPUs []struct {
|
||||
ID string `xml:"id,attr"`
|
||||
PCI struct {
|
||||
Replay string `xml:"replay_counter"`
|
||||
} `xml:"pci"`
|
||||
Retired struct {
|
||||
SBE string `xml:"multiple_single_bit_retirement>retired_count"`
|
||||
DBE string `xml:"double_bit_retirement>retired_count"`
|
||||
Pending string `xml:"pending_retirement"`
|
||||
// Older drivers name it pending_blacklist.
|
||||
Blacklist string `xml:"pending_blacklist"`
|
||||
} `xml:"retired_pages"`
|
||||
Remapped struct {
|
||||
Corr string `xml:"remapped_row_corr"`
|
||||
Unc string `xml:"remapped_row_unc"`
|
||||
Pending string `xml:"remapped_row_pending"`
|
||||
Failure string `xml:"remapped_row_failure"`
|
||||
} `xml:"remapped_rows"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseHealthReport reads each card's health, in index order, and maps each
|
||||
// card's PCI bus ("3b:00", as the kernel names it) to its index.
|
||||
func parseHealthReport(report []byte) ([]GPUWear, map[string]int) {
|
||||
var log healthLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
bus := map[string]int{}
|
||||
out := make([]GPUWear, 0, len(log.GPUs))
|
||||
for i, g := range log.GPUs {
|
||||
bus[busOf(g.ID)] = i
|
||||
out = append(out, GPUWear{
|
||||
Index: i,
|
||||
PCIeReplays: count(g.PCI.Replay),
|
||||
RetiredSBE: count(g.Retired.SBE),
|
||||
RetiredDBE: count(g.Retired.DBE),
|
||||
RetirePending: yes(g.Retired.Pending) || yes(g.Retired.Blacklist),
|
||||
RemappedCorrectable: count(g.Remapped.Corr),
|
||||
RemappedUncorrectable: count(g.Remapped.Unc),
|
||||
RemapPending: yes(g.Remapped.Pending),
|
||||
RemapFailure: yes(g.Remapped.Failure),
|
||||
})
|
||||
}
|
||||
return out, bus
|
||||
}
|
||||
|
||||
// busOf is a PCI address as the kernel's XID lines carry it: nvidia-smi
|
||||
// says "00000000:3B:00.0", the kernel "0000:3b:00".
|
||||
func busOf(id string) string {
|
||||
id = strings.ToLower(strings.TrimSpace(id))
|
||||
if i := strings.LastIndex(id, "."); i >= 0 {
|
||||
id = id[:i]
|
||||
}
|
||||
parts := strings.Split(id, ":")
|
||||
if len(parts) >= 2 {
|
||||
return parts[len(parts)-2] + ":" + parts[len(parts)-1]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func count(s string) int64 {
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 64)
|
||||
if err != nil {
|
||||
return -1
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func yes(s string) bool { return strings.EqualFold(strings.TrimSpace(s), "yes") }
|
||||
|
||||
// nvlinkError matches a line of `nvidia-smi nvlink -e`, whose wording varies
|
||||
// by driver: "Link 0: Replay Errors: 0", "Link 0: CRC Errors: 0",
|
||||
// "Link 3: Data CRC Errors: 2".
|
||||
var (
|
||||
nvlinkGPU = regexp.MustCompile(`^GPU (\d+):`)
|
||||
nvlinkError = regexp.MustCompile(`Link \d+:\s*(.*?)\s*Errors?\s*:\s*(\d+)`)
|
||||
)
|
||||
|
||||
// parseNVLinkErrors sums each card's link errors by kind: CRC, replay,
|
||||
// recovery.
|
||||
func parseNVLinkErrors(out string) map[int][3]int64 {
|
||||
res := map[int][3]int64{}
|
||||
gpu := -1
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if m := nvlinkGPU.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
|
||||
gpu, _ = strconv.Atoi(m[1])
|
||||
continue
|
||||
}
|
||||
m := nvlinkError.FindStringSubmatch(line)
|
||||
if m == nil || gpu < 0 {
|
||||
continue
|
||||
}
|
||||
n, _ := strconv.ParseInt(m[2], 10, 64)
|
||||
e := res[gpu]
|
||||
kind := strings.ToLower(m[1])
|
||||
switch {
|
||||
case strings.Contains(kind, "crc"):
|
||||
e[0] += n
|
||||
case strings.Contains(kind, "replay"):
|
||||
e[1] += n
|
||||
case strings.Contains(kind, "recovery"):
|
||||
e[2] += n
|
||||
}
|
||||
res[gpu] = e
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
type xidEvent struct {
|
||||
bus string
|
||||
code int
|
||||
text string
|
||||
at float64
|
||||
}
|
||||
|
||||
// xidLine matches the driver's report of an XID:
|
||||
// "NVRM: Xid (PCI:0000:3b:00): 79, pid=..., GPU has fallen off the bus."
|
||||
var xidLine = regexp.MustCompile(`NVRM: Xid \(PCI:([0-9a-fA-F:.]+)\): (\d+),\s*(.*)$`)
|
||||
|
||||
// parseXIDs reads journalctl's short-unix lines: a timestamp, then the
|
||||
// message.
|
||||
func parseXIDs(out string) []xidEvent {
|
||||
var evs []xidEvent
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
m := xidLine.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
code, _ := strconv.Atoi(m[2])
|
||||
e := xidEvent{bus: busOf(m[1]), code: code, text: strings.TrimSpace(m[3])}
|
||||
if f := strings.Fields(line); len(f) > 0 {
|
||||
e.at, _ = strconv.ParseFloat(f[0], 64)
|
||||
}
|
||||
evs = append(evs, e)
|
||||
}
|
||||
return evs
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const healthXML = `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<gpu id="00000000:3B:00.0">
|
||||
<pci><replay_counter>0</replay_counter></pci>
|
||||
<retired_pages>
|
||||
<multiple_single_bit_retirement><retired_count>N/A</retired_count></multiple_single_bit_retirement>
|
||||
<double_bit_retirement><retired_count>N/A</retired_count></double_bit_retirement>
|
||||
<pending_retirement>N/A</pending_retirement>
|
||||
</retired_pages>
|
||||
<remapped_rows>
|
||||
<remapped_row_corr>2</remapped_row_corr>
|
||||
<remapped_row_unc>1</remapped_row_unc>
|
||||
<remapped_row_pending>Yes</remapped_row_pending>
|
||||
<remapped_row_failure>No</remapped_row_failure>
|
||||
</remapped_rows>
|
||||
</gpu>
|
||||
<gpu id="00000000:5E:00.0">
|
||||
<pci><replay_counter>4021</replay_counter></pci>
|
||||
<retired_pages>
|
||||
<multiple_single_bit_retirement><retired_count>3</retired_count></multiple_single_bit_retirement>
|
||||
<double_bit_retirement><retired_count>1</retired_count></double_bit_retirement>
|
||||
<pending_blacklist>Yes</pending_blacklist>
|
||||
</retired_pages>
|
||||
<remapped_rows>N/A</remapped_rows>
|
||||
</gpu>
|
||||
</nvidia_smi_log>`
|
||||
|
||||
const nvlinkOut = `GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-aaa)
|
||||
Link 0: Replay Errors: 1
|
||||
Link 0: Recovery Errors: 0
|
||||
Link 0: CRC Errors: 5
|
||||
Link 1: Data CRC Errors: 2
|
||||
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-bbb)
|
||||
Link 0: Replay Errors: 0
|
||||
`
|
||||
|
||||
func TestAGPUsWearAndFailuresAreReadAndCountedFromWhenTheAgentStarted(t *testing.T) {
|
||||
now := time.Unix(1_800_000_000, 0)
|
||||
journal := ""
|
||||
h := &HealthSampler{now: func() time.Time { return now }, xids: map[int]*GPUWear{},
|
||||
run: func(_ context.Context, name string, args ...string) ([]byte, error) {
|
||||
switch {
|
||||
case name == "nvidia-smi" && args[0] == "-q":
|
||||
return []byte(healthXML), nil
|
||||
case name == "nvidia-smi" && args[0] == "nvlink":
|
||||
return []byte(nvlinkOut), nil
|
||||
case name == "journalctl":
|
||||
if journal == "" {
|
||||
return nil, errors.New("exit status 1")
|
||||
}
|
||||
return []byte(journal), nil
|
||||
}
|
||||
return nil, errors.New("unexpected " + name)
|
||||
}}
|
||||
got := h.Sample(t.Context())
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
a, b := got[0], got[1]
|
||||
if a.RemappedCorrectable != 2 || a.RemappedUncorrectable != 1 || !a.RemapPending || a.RemapFailure ||
|
||||
a.RetiredSBE != -1 || a.PCIeReplays != 0 || a.NVLinkCRC != 7 || a.NVLinkReplay != 1 {
|
||||
t.Errorf("card 0 %+v", a)
|
||||
}
|
||||
if b.PCIeReplays != 4021 || b.RetiredSBE != 3 || b.RetiredDBE != 1 || !b.RetirePending || b.RemappedCorrectable != -1 {
|
||||
t.Errorf("card 1 %+v", b)
|
||||
}
|
||||
|
||||
// An XID on card 1, logged after the agent started; read on the next
|
||||
// sample a minute on.
|
||||
journal = "1800000030.123456 gpu07 kernel: NVRM: Xid (PCI:0000:5e:00): 79, pid=1234, name=llama-server, GPU has fallen off the bus.\n"
|
||||
now = now.Add(61 * time.Second)
|
||||
got = h.Sample(t.Context())
|
||||
if got[1].XIDs != 1 || got[1].LastXID != 79 || !strings.Contains(got[1].LastXIDText, "fallen off the bus") ||
|
||||
got[1].LastXIDAt.Unix() != 1_800_000_030 || got[0].XIDs != 0 {
|
||||
t.Errorf("after an XID: %+v / %+v", got[0], got[1])
|
||||
}
|
||||
// Within the minute, the last reading.
|
||||
journal = ""
|
||||
now = now.Add(10 * time.Second)
|
||||
if again := h.Sample(t.Context()); again[1].XIDs != 1 {
|
||||
t.Errorf("%+v", again[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestBusIDsAreReadAsTheKernelWritesThem(t *testing.T) {
|
||||
for in, want := range map[string]string{"00000000:3B:00.0": "3b:00", "0000:3b:00": "3b:00", "0000:5e:00.0": "5e:00"} {
|
||||
if got := busOf(in); got != want {
|
||||
t.Errorf("%s: %s, want %s", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
@@ -10,10 +11,14 @@ import (
|
||||
|
||||
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
|
||||
// nvidia-smi index: what a process actually computes on, as opposed to what it
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose CUDA
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
|
||||
// backend did not load runs on the CPU without a word, and this is where that
|
||||
// shows — the process is on no GPU's list.
|
||||
//
|
||||
// From the full query rather than --query-compute-apps, which lists compute
|
||||
// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
|
||||
// would have read as not on its GPU at all.
|
||||
//
|
||||
// A node with no nvidia-smi returns nil and no error, as discovery does.
|
||||
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
if binary == "" {
|
||||
@@ -22,44 +27,38 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
}
|
||||
binary = "nvidia-smi"
|
||||
}
|
||||
gpus, err := exec.CommandContext(ctx, binary, "--query-gpu=index,uuid", "--format=csv,noheader").Output()
|
||||
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
apps, err := exec.CommandContext(ctx, binary, "--query-compute-apps=gpu_uuid,pid", "--format=csv,noheader").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseGPUProcesses(string(gpus), string(apps)), nil
|
||||
return parseGPUProcesses(out)
|
||||
}
|
||||
|
||||
// parseGPUProcesses joins nvidia-smi's GPU list (index, uuid) to its compute
|
||||
// apps (gpu_uuid, pid): the apps are listed by UUID, placement speaks index.
|
||||
func parseGPUProcesses(gpus, apps string) map[int][]int {
|
||||
index := map[string]int{}
|
||||
for _, line := range strings.Split(gpus, "\n") {
|
||||
i, uuid, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
n, err := strconv.Atoi(strings.TrimSpace(i))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
index[strings.TrimSpace(uuid)] = n
|
||||
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
|
||||
// index order, and the processes on it.
|
||||
type smiLog struct {
|
||||
GPUs []struct {
|
||||
Processes []struct {
|
||||
PID string `xml:"pid"`
|
||||
Type string `xml:"type"`
|
||||
} `xml:"processes>process_info"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseGPUProcesses reads the report. GPUs are listed in index order, which
|
||||
// is nvidia-smi's own numbering — the one placement assigns by.
|
||||
func parseGPUProcesses(report []byte) (map[int][]int, error) {
|
||||
var log smiLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
out := map[int][]int{}
|
||||
for _, line := range strings.Split(apps, "\n") {
|
||||
uuid, p, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
for i, g := range log.GPUs {
|
||||
for _, p := range g.Processes {
|
||||
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
|
||||
out[i] = append(out[i], pid)
|
||||
}
|
||||
}
|
||||
pid, err := strconv.Atoi(strings.TrimSpace(p))
|
||||
i, known := index[strings.TrimSpace(uuid)]
|
||||
if err != nil || !known {
|
||||
continue
|
||||
}
|
||||
out[i] = append(out[i], pid)
|
||||
}
|
||||
return out
|
||||
return out, nil
|
||||
}
|
||||
@@ -5,17 +5,34 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGPUProcessesAreListedByIndexNotUUID(t *testing.T) {
|
||||
gpus := "0, GPU-aaa\n1, GPU-bbb\n2, GPU-ccc\n"
|
||||
// Two processes on GPU 1, one on GPU 2, and a GPU this list has never
|
||||
// heard of, which is dropped rather than guessed at.
|
||||
apps := "GPU-bbb, 4100\nGPU-ccc, 4200\nGPU-bbb, 4300\nGPU-zzz, 9999\n"
|
||||
got := parseGPUProcesses(gpus, apps)
|
||||
func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
|
||||
report := `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<driver_version>580.65.06</driver_version>
|
||||
<gpu id="00000000:17:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes></processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:65:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes>
|
||||
<process_info><pid>4100</pid><type>C</type><process_name>llama-server</process_name></process_info>
|
||||
<process_info><pid>4300</pid><type>G</type><process_name>llama-server</process_name></process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:B3:00.0"><processes>
|
||||
<process_info><pid>4200</pid><type>C+G</type></process_info>
|
||||
</processes></gpu>
|
||||
</nvidia_smi_log>`
|
||||
got, err := parseGPUProcesses([]byte(report))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A Vulkan engine is a graphics process ("G") and is on its GPU all the
|
||||
// same; a compute-only query would have missed it.
|
||||
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
if got := parseGPUProcesses(gpus, ""); len(got) != 0 {
|
||||
t.Errorf("no compute apps should be no GPU in use: %v", got)
|
||||
if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
|
||||
t.Error("a report that does not parse was read as no processes")
|
||||
}
|
||||
}
|
||||
@@ -41,6 +41,14 @@ type GPUSensor struct {
|
||||
// is running free. Idle and application clock settings are left out;
|
||||
// they are not the card being held back.
|
||||
Throttle []string
|
||||
// How busy it is, read with the rest every sample. Discovery reads the
|
||||
// same once, at start, which made a busy card look as idle as it was
|
||||
// when the agent came up. -1 is not reported by the driver; zero
|
||||
// MemoryTotalMB is a sample without these at all.
|
||||
UtilPercent, MemUtilPercent float64
|
||||
MemoryUsedMB, MemoryTotalMB float64
|
||||
SMClockMHz float64
|
||||
ECCUncorrected float64
|
||||
}
|
||||
|
||||
// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
|
||||
@@ -187,13 +195,33 @@ var throttleBits = []struct {
|
||||
// gpus asks nvidia-smi for each card's heat and power. The memory temperature
|
||||
// is not known to every driver, and a field nvidia-smi does not know fails the
|
||||
// whole query, so it is asked for first and dropped on failure.
|
||||
// gpuFields are what each sample asks nvidia-smi for, in order. The busy
|
||||
// figures follow the throttle reasons; the memory temperature is last, being
|
||||
// the one field a driver may reject outright.
|
||||
const (
|
||||
gpuBase = "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active"
|
||||
gpuBusy = ",utilization.gpu,utilization.memory,memory.used,memory.total,clocks.sm,ecc.errors.uncorrected.volatile.total"
|
||||
gpuHBMTC = ",temperature.memory"
|
||||
)
|
||||
|
||||
func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
|
||||
withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
|
||||
out, err := s.smi(ctx, withMem)
|
||||
if err != nil {
|
||||
if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
|
||||
return nil
|
||||
// Most fields first, fewer on a driver that refuses one: a query naming
|
||||
// a field the driver does not know fails as a whole.
|
||||
var out []byte
|
||||
var busy bool
|
||||
for _, q := range []struct {
|
||||
fields string
|
||||
busy bool
|
||||
}{{gpuBase + gpuBusy + gpuHBMTC, true}, {gpuBase + gpuBusy, true}, {gpuBase + gpuHBMTC, false}, {gpuBase, false}} {
|
||||
var err error
|
||||
if out, err = s.smi(ctx, q.fields); err == nil {
|
||||
busy = q.busy
|
||||
break
|
||||
}
|
||||
out = nil
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
var gs []GPUSensor
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
@@ -216,14 +244,31 @@ func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(f) > 5 {
|
||||
g.MemoryTempC = num(f[5])
|
||||
rest := f[5:]
|
||||
if busy && len(rest) >= 6 {
|
||||
g.UtilPercent, g.MemUtilPercent = numOrUnknown(rest[0]), numOrUnknown(rest[1])
|
||||
g.MemoryUsedMB, g.MemoryTotalMB = num(rest[2]), num(rest[3])
|
||||
g.SMClockMHz, g.ECCUncorrected = num(rest[4]), numOrUnknown(rest[5])
|
||||
rest = rest[6:]
|
||||
}
|
||||
if len(rest) > 0 {
|
||||
g.MemoryTempC = num(rest[0])
|
||||
}
|
||||
gs = append(gs, g)
|
||||
}
|
||||
return gs
|
||||
}
|
||||
|
||||
// numOrUnknown is a figure, or -1 where nvidia-smi says it has none: a card
|
||||
// without ECC has no error count, which is not the same as none.
|
||||
func numOrUnknown(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil || v < 0 {
|
||||
return -1
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
|
||||
func num(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
|
||||
@@ -167,3 +167,44 @@ func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T
|
||||
t.Errorf("%+v", got.GPUs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachSampleSaysHowBusyEachGPUIs(t *testing.T) {
|
||||
// What the card is doing now, every sample: discovery read it once, at
|
||||
// start, and a card busy since looked as idle as it was then.
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
if !strings.Contains(fields, "utilization.gpu") {
|
||||
return nil, errors.New("asked without how busy the cards are")
|
||||
}
|
||||
return []byte(
|
||||
"0, 71, 612.45, 700.00, 0x0000000000000000, 97, 64, 71234, 81559, 1980, 0, 84\n" +
|
||||
"1, 30, 70.10, 700.00, 0x0000000000000001, 0, 0, 512, 81559, 345, [N/A], [N/A]\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
g0, g1 := got[0], got[1]
|
||||
if g0.UtilPercent != 97 || g0.MemUtilPercent != 64 || g0.MemoryUsedMB != 71234 || g0.MemoryTotalMB != 81559 ||
|
||||
g0.SMClockMHz != 1980 || g0.ECCUncorrected != 0 || g0.MemoryTempC != 84 {
|
||||
t.Errorf("card 0 %+v", g0)
|
||||
}
|
||||
// Idle is zero; a count the card does not keep is unknown, not zero.
|
||||
if g1.UtilPercent != 0 || g1.ECCUncorrected != -1 || g1.MemoryTempC != 0 {
|
||||
t.Errorf("card 1 %+v", g1)
|
||||
}
|
||||
}
|
||||
|
||||
func TestADriverThatDoesNotKnowTheBusyFieldsStillReportsHeatAndPower(t *testing.T) {
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
if strings.Contains(fields, "utilization.gpu") {
|
||||
return nil, errors.New(`Field "ecc.errors.uncorrected.volatile.total" is not a valid field to query.`)
|
||||
}
|
||||
return []byte("0, 66, 250.0, 300.0, 0x0000000000000000, 70\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
|
||||
if len(got) != 1 || got[0].TempC != 66 || got[0].MemoryTempC != 70 || got[0].MemoryTotalMB != 0 {
|
||||
t.Errorf("%+v", got)
|
||||
}
|
||||
}
|
||||
@@ -168,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if s.APIKey != "" {
|
||||
env["LLAMA_API_KEY"] = s.APIKey
|
||||
}
|
||||
// The Vulkan build — what a GPU older than the CUDA builds gets — does
|
||||
// not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
|
||||
// as the driver lists them, which on NVIDIA follows the same bus order
|
||||
// as nvidia-smi; were it not to, the shard would not hold memory on its
|
||||
// own GPUs, and starting it checks exactly that.
|
||||
if len(s.GPUIndices) > 0 {
|
||||
env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
|
||||
}
|
||||
return []Command{{
|
||||
Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
|
||||
}}, nil
|
||||
|
||||
@@ -143,7 +143,7 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, w.Variant, in.Host)
|
||||
builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Unsupported, err.Error()
|
||||
var nh NotHere
|
||||
@@ -152,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
}
|
||||
return st
|
||||
}
|
||||
st.Variant = build.Variant
|
||||
dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
|
||||
|
||||
// Installed before: check it still runs, and trust nothing else about it.
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
|
||||
// Installed before: whichever candidate is, best first, checked that it
|
||||
// still runs and trusted for nothing else.
|
||||
for _, build := range builds {
|
||||
dir := in.dirFor(w, build)
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
|
||||
continue
|
||||
}
|
||||
st := st.as(w, build)
|
||||
// The packages are checked too. Removing one is exactly the kind of
|
||||
// change an unrelated upgrade makes, and ldd below would catch it only
|
||||
// as a missing library with no hint of which package held it.
|
||||
// change an unrelated upgrade makes, and ldd below would catch it
|
||||
// only as a missing library with no hint of which package held it.
|
||||
if err := in.packages(ctx, build.Packages, &st); err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
@@ -169,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
in.log().Warn("an installed engine no longer runs; installing it again",
|
||||
"engine", w.Engine, "version", w.Version, "err", err)
|
||||
"engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
|
||||
break
|
||||
}
|
||||
|
||||
if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
|
||||
st.State, st.Detail, st.Binary = Failed, err.Error(), ""
|
||||
return st
|
||||
// Each candidate in turn until one installs and is seen to work: a build
|
||||
// the table was wrong about — a driver too old for it after all, a GPU
|
||||
// it has no kernels for — costs its download, and the next is tried.
|
||||
var tried []string
|
||||
for _, build := range builds {
|
||||
st := st.as(w, build)
|
||||
if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
in.log().Warn("an engine build did not work here; trying the next",
|
||||
"engine", w.Engine, "variant", build.Variant, "err", err)
|
||||
continue
|
||||
}
|
||||
bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
|
||||
if err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
bin, version, err := in.verify(ctx, dir, build)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
st.State, st.Binary = Failed, ""
|
||||
st.Detail = strings.Join(tried, "; then ")
|
||||
if len(builds) == 1 {
|
||||
_, st.Detail, _ = strings.Cut(st.Detail, ": ")
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
return st
|
||||
}
|
||||
|
||||
// dirFor is where a build of an engine is installed: by the version it is
|
||||
// and its variant, so two builds of one version sit side by side.
|
||||
func (in *Installer) dirFor(w Want, b Build) string {
|
||||
return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
|
||||
}
|
||||
|
||||
// versionOr is the version a build is: its own, when it is not the one asked
|
||||
// for.
|
||||
func (b Build) versionOr(asked string) string {
|
||||
if b.Version != "" {
|
||||
return b.Version
|
||||
}
|
||||
return asked
|
||||
}
|
||||
|
||||
// as is a status for one build of what was asked for.
|
||||
func (st Status) as(w Want, b Build) Status {
|
||||
st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
|
||||
return st
|
||||
}
|
||||
|
||||
func (in *Installer) ready(st Status, b Build, bin, version string) Status {
|
||||
st.State, st.Binary = Ready, bin
|
||||
st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
|
||||
if b.Note != "" {
|
||||
st.Detail += "; " + b.Note
|
||||
}
|
||||
var missing []string
|
||||
for _, f := range b.CPUFlags {
|
||||
if !in.Host.HasCPUFlag(f) {
|
||||
|
||||
@@ -255,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
|
||||
func TestTheBuildFollowsTheHardware(t *testing.T) {
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
b, err := chooseLlamaCpp("b1", "", gpu)
|
||||
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
|
||||
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
|
||||
got, err := llamaCppCandidates("b1", "", gpu)
|
||||
if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
|
||||
t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
|
||||
}
|
||||
if _, err := chooseLlamaCpp("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
|
||||
}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
host discovery.Host
|
||||
want []string
|
||||
}{
|
||||
// Driver 580 and Turing: CUDA 13.3 first.
|
||||
{"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
|
||||
// A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
|
||||
{"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
|
||||
// An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
|
||||
{"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
|
||||
// A Pascal card: CUDA 12.8, not 13.3.
|
||||
{"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
|
||||
// arm64 with a GPU: never the CPU build.
|
||||
{"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
|
||||
} {
|
||||
got, err := llamaCppCandidates("b1", "", c.host)
|
||||
var names []string
|
||||
for _, b := range got {
|
||||
names = append(names, b.Variant)
|
||||
}
|
||||
if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
|
||||
t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
|
||||
}
|
||||
}
|
||||
|
||||
slow := ubuntu
|
||||
slow.CPUFlags = nil
|
||||
@@ -400,7 +425,7 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
|
||||
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
|
||||
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
|
||||
// loader looks, and llama-server serving a GPU placement on the CPU.
|
||||
cuda := "bin-ubuntu-cuda-" + cudaForLlamaCpp + "-x64.tar.gz"
|
||||
cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
|
||||
rel := newRelease(map[string][]byte{
|
||||
"llama-b1-" + cuda: tarGz(t, map[string]string{
|
||||
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
|
||||
|
||||
@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
|
||||
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
|
||||
defer cancel()
|
||||
// The version this build is, which for a GPU older than every build of
|
||||
// the one asked for is an older one: its publication date is the one
|
||||
// the dependencies are resolved as of.
|
||||
w.Version = b.versionOr(w.Version)
|
||||
|
||||
if err := in.packages(ctx, b.Packages, st); err != nil {
|
||||
return err
|
||||
@@ -91,17 +95,22 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
// refuses a download that does not match it.
|
||||
in.progress(st, "asking GitHub for %s's digest", b.Wheel.Asset)
|
||||
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum, ok := sums[b.Wheel.Asset]
|
||||
if !ok {
|
||||
return fmt.Errorf("%s %s publishes no %s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Asset)
|
||||
var asset, sum string
|
||||
for name, s := range sums {
|
||||
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
|
||||
asset, sum = name, s
|
||||
}
|
||||
}
|
||||
if asset == "" {
|
||||
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
|
||||
b.Wheel.Tag + "/" + b.Wheel.Asset
|
||||
b.Wheel.Tag + "/" + asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
|
||||
@@ -25,8 +25,9 @@ type fakePython struct {
|
||||
commands []string
|
||||
failPip bool
|
||||
// noKernel makes the GPU check fail as a torch without kernels for the
|
||||
// card does.
|
||||
noKernel bool
|
||||
// card does: in every build, or in noKernelIn only ("-cu13.").
|
||||
noKernel bool
|
||||
noKernelIn string
|
||||
}
|
||||
|
||||
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
@@ -48,7 +49,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
|
||||
if f.noKernel {
|
||||
if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
|
||||
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
|
||||
}
|
||||
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
|
||||
@@ -77,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
|
||||
{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
|
||||
})
|
||||
mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
|
||||
})
|
||||
pypi := httptest.NewServer(mux)
|
||||
t.Cleanup(pypi.Close)
|
||||
return &Installer{
|
||||
@@ -143,7 +147,8 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.0 or newer"},
|
||||
// No vLLM was ever built for Pascal.
|
||||
{"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -195,15 +200,16 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderGPUGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A V100: CUDA 13 dropped it. The same vLLM's CUDA 12.9 build, from the
|
||||
// project's GitHub release, with the torch PyTorch publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
|
||||
// 12.9 build, from the project's GitHub release, with the torch PyTorch
|
||||
// publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, gh := pythonInstaller(t, volta, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("a V100 node: %+v", got[0])
|
||||
t.Fatalf("a Turing node on driver 550: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
|
||||
t.Errorf("not installed under its own build's name: %v", err)
|
||||
@@ -236,11 +242,7 @@ func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
|
||||
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
|
||||
}
|
||||
volta := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}
|
||||
if _, err := chooseVLLM("0.30.0", "cu13", volta); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu13 chosen for a V100 should be refused, saying why: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu200", gpuNode); err == nil {
|
||||
if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
|
||||
t.Error("a build that does not exist was accepted")
|
||||
}
|
||||
}
|
||||
@@ -260,11 +262,51 @@ func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
|
||||
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
// A machine vLLM is not published for is refused, not given the other
|
||||
// build to fail with.
|
||||
if _, err := chooseVLLM("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
!strings.Contains(err.Error(), "no Linux build") {
|
||||
t.Errorf("an unknown architecture: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
// Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
|
||||
got, err := vllmCandidates("0.30.0", "", gpuNode)
|
||||
if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
|
||||
t.Errorf("candidates on a current node: %+v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
|
||||
// 0.30.0 asked for; no build of it has Volta kernels. The node installs
|
||||
// 0.19.1, the newest that has, and says so.
|
||||
v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, v100, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
|
||||
!strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
|
||||
t.Fatalf("a V100: %+v", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
|
||||
t.Errorf("not installed as what it is: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
|
||||
t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
|
||||
}
|
||||
// Chosen cu13, which 0.19.1 has none of: refused, saying why.
|
||||
if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
|
||||
t.Error("a CUDA 13 build was offered to a V100")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
|
||||
// The table says CUDA 13 runs here; the card says otherwise. The next
|
||||
// build is installed rather than the node going without.
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Variant != "cu129" {
|
||||
t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
|
||||
}
|
||||
}
|
||||
+309
-224
@@ -10,7 +10,6 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -60,15 +59,23 @@ type Build struct {
|
||||
// with no kernels for the card — a GPU generation its torch dropped —
|
||||
// imports and prints its version, and fails on its first model.
|
||||
GPUCheck bool
|
||||
// Version is the engine version this build is, when it is not the one
|
||||
// asked for: the newest built for this machine's GPUs, where the one
|
||||
// asked for has no kernels for them. Empty is the version asked for.
|
||||
Version string
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset.
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
Project, Repo, Tag, Asset string
|
||||
Project, Repo, Tag, Prefix, Suffix string
|
||||
}
|
||||
|
||||
// Recipe knows how to install one engine.
|
||||
@@ -78,10 +85,12 @@ type Recipe struct {
|
||||
Repo string
|
||||
// PyPI is the project an engine installed from PyPI is published as.
|
||||
PyPI string
|
||||
// Choose picks the build for a machine, or says why none fits. variant
|
||||
// is a build chosen for the node ("cu129"); empty leaves it to the
|
||||
// machine.
|
||||
Choose func(version, variant string, h discovery.Host) (Build, error)
|
||||
// Candidates are the builds that may run on a machine, best first, or
|
||||
// why none can. The installer takes the first that installs and is seen
|
||||
// to work — computing on the GPU, where there is one — so a build this
|
||||
// table is wrong about costs a download, not a node. variant is a build
|
||||
// chosen for the node; empty leaves the choice to the machine.
|
||||
Candidates func(version, variant string, h discovery.Host) ([]Build, error)
|
||||
}
|
||||
|
||||
// Recipes are the engines this agent can install.
|
||||
@@ -90,10 +99,10 @@ type Recipe struct {
|
||||
// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
|
||||
// checked against its published checksum.
|
||||
var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
|
||||
"vllm": {PyPI: "vllm", Choose: chooseVLLM},
|
||||
"sglang": {PyPI: "sglang", Choose: chooseSGLang},
|
||||
"uv": {Repo: "astral-sh/uv", Choose: chooseUV},
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
// Installable are the engines this agent can install, as it tells the
|
||||
@@ -123,7 +132,15 @@ type NotHere struct{ Reason string }
|
||||
|
||||
func (e NotHere) Error() string { return e.Reason }
|
||||
|
||||
func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
|
||||
b, err := chooseUV(version, h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []Build{b}, nil
|
||||
}
|
||||
|
||||
func chooseUV(_ string, h discovery.Host) (Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -140,32 +157,190 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
|
||||
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
|
||||
const minDriverForCUDA13 = 580
|
||||
|
||||
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
|
||||
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
|
||||
// version compatibility, and on Volta.
|
||||
// What builds need of the NVIDIA side of a machine, read from what each
|
||||
// project publishes and how it was built:
|
||||
//
|
||||
// - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
|
||||
// by CUDA's minor version compatibility — except code shipped only as
|
||||
// PTX, which the driver compiles and so needs a driver as new as the
|
||||
// CUDA it was compiled with.
|
||||
// - Which GPUs a build has kernels for is the build's own list, not what
|
||||
// its CUDA supports.
|
||||
const (
|
||||
minDriverForCUDA12 = 525
|
||||
minComputeForCUDA12 = 7.0
|
||||
minDriverForCUDA13 = 580
|
||||
minDriverForCUDA12 = 525
|
||||
)
|
||||
|
||||
// cuda13Since is the first version of each Python engine built for CUDA 13.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
|
||||
// gpuNeeds is what one build needs of a machine's GPUs and driver.
|
||||
type gpuNeeds struct {
|
||||
driver int
|
||||
cc float64
|
||||
}
|
||||
|
||||
// cudaGate says whether a node can run this version of an engine, and which
|
||||
// CUDA it is built for.
|
||||
// fits says why a machine cannot run a build, or nil.
|
||||
func (n gpuNeeds) fits(what string, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
|
||||
return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
|
||||
return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// candidate is a build and what it needs.
|
||||
type candidate struct {
|
||||
b Build
|
||||
needs gpuNeeds
|
||||
what string
|
||||
}
|
||||
|
||||
// pick keeps the candidates a machine can run, in order, and those a
|
||||
// variant chosen for the node names; with none left, it says why each went.
|
||||
func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
|
||||
var out []Build
|
||||
var why []string
|
||||
for _, c := range cands {
|
||||
if variant != "" && c.b.Variant != variant {
|
||||
continue
|
||||
}
|
||||
if err := c.needs.fits(c.what, h); err != nil {
|
||||
why = append(why, err.Error())
|
||||
continue
|
||||
}
|
||||
out = append(out, c.b)
|
||||
}
|
||||
switch {
|
||||
case len(out) > 0:
|
||||
return out, nil
|
||||
case len(why) == 0 && variant != "":
|
||||
var names []string
|
||||
for _, c := range cands {
|
||||
names = append(names, c.b.Variant)
|
||||
}
|
||||
return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
|
||||
}
|
||||
return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
|
||||
}
|
||||
|
||||
// --- vLLM ---
|
||||
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for drivers CUDA 13 does not run on.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
|
||||
// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
|
||||
// "+cu129" beside a CUDA 13 one.
|
||||
const vllmCUDA13Since = "0.20"
|
||||
|
||||
// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
|
||||
// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
|
||||
// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
|
||||
// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
|
||||
// 0.20 and 8.0 since.
|
||||
func vllmMinCompute(version, arch string) float64 {
|
||||
old := versionBefore(version, vllmCUDA13Since)
|
||||
switch {
|
||||
case arch == "arm64" && old:
|
||||
return 8.7
|
||||
case arch == "arm64":
|
||||
return 8.0
|
||||
case old:
|
||||
return 7.0
|
||||
}
|
||||
return 7.5
|
||||
}
|
||||
|
||||
// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
|
||||
// on an architecture, for GPUs older than every build of the version asked
|
||||
// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
|
||||
// for a GPU no vLLM was built for.
|
||||
var vllmNewestFor = []struct {
|
||||
arch string
|
||||
from float64 // the oldest GPU it covers
|
||||
version string
|
||||
}{
|
||||
{"amd64", 7.0, "0.19.1"},
|
||||
}
|
||||
|
||||
// vllmCandidates are the vLLM builds for a machine.
|
||||
//
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
|
||||
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older build (engine_overrides in the server's configuration)
|
||||
// rather than going without vLLM, or without the Volta nodes.
|
||||
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
|
||||
// The version asked for, in the build the machine runs: CUDA 13 where the
|
||||
// driver allows, the same version's CUDA 12.9 build where it does not. One
|
||||
// version across the cluster wherever the GPUs allow it, because the flags
|
||||
// llmgw builds are checked against it. Where they do not — GPUs older than
|
||||
// every build of that version — the newest version built for them, said so
|
||||
// beside the installation: vLLM on a V100 at all, rather than not.
|
||||
func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
arch, err := gpuArch("vLLM", h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v, note := version, ""
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
|
||||
older := ""
|
||||
for _, n := range vllmNewestFor {
|
||||
if n.arch == h.Arch && cc >= n.from {
|
||||
older = n.version
|
||||
}
|
||||
}
|
||||
if older == "" {
|
||||
return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
|
||||
"the node serves llama.cpp models", cc, arch)
|
||||
}
|
||||
v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
|
||||
}
|
||||
pypi := func(variant string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + v},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
Note: note,
|
||||
}
|
||||
if v != version {
|
||||
b.Version = v
|
||||
}
|
||||
return b
|
||||
}
|
||||
minCC := vllmMinCompute(v, h.Arch)
|
||||
if versionBefore(v, vllmCUDA13Since) {
|
||||
// PyPI's own build is CUDA 12, and is the one there is; asked for
|
||||
// cu129, this is it.
|
||||
if variant == "cu129" {
|
||||
variant = "cu12"
|
||||
}
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
|
||||
}, h)
|
||||
}
|
||||
cu129 := pypi("cu129")
|
||||
cu129.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
cu129.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
|
||||
Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
|
||||
}
|
||||
cu129.ExtraIndex = pytorchCU129
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
|
||||
}, h)
|
||||
}
|
||||
|
||||
// gpuArch says whether a Python GPU engine is published for this machine
|
||||
// and whether it has an NVIDIA GPU at all.
|
||||
func gpuArch(engine string, h discovery.Host) (string, error) {
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
@@ -174,25 +349,42 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
|
||||
if h.NVIDIADriver == "" {
|
||||
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
|
||||
if versionBefore(version, cuda13Since[engine]) {
|
||||
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
hint := ""
|
||||
if cuda == "13" {
|
||||
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
|
||||
}
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, hint)
|
||||
}
|
||||
return variant, nil
|
||||
return h.Arch, nil
|
||||
}
|
||||
|
||||
// --- SGLang ---
|
||||
|
||||
// sglangCUDA13Since is the first SGLang built for CUDA 13.
|
||||
const sglangCUDA13Since = "0.5"
|
||||
|
||||
func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("SGLang", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
b := Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
needs := gpuNeeds{minDriverForCUDA13, 7.5}
|
||||
if versionBefore(version, sglangCUDA13Since) {
|
||||
b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
|
||||
}
|
||||
return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
// launcher for its kernels the first time a model loads, and without gcc
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
@@ -216,143 +408,6 @@ func versionBefore(a, b string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
// launcher for its kernels the first time a model loads, and without gcc
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for the drivers and GPUs CUDA 13 left behind.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// chooseVLLM picks the vLLM build for a machine.
|
||||
//
|
||||
// Left to the machine (variant ""), it is CUDA 13 where that runs, and the
|
||||
// same version's CUDA 12.9 build where only that does: one version across
|
||||
// the cluster, which matters because the flags llmgw builds are checked
|
||||
// against one version, and an older vLLM on the older cards would be a
|
||||
// second set of flags nobody checked. A variant chosen for the node is that
|
||||
// build, if the node can run it.
|
||||
func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
// What no build of it gets past: a machine it is not published for, or
|
||||
// one with no NVIDIA GPU. Only a CUDA requirement falls back to the
|
||||
// CUDA 12.9 build.
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return Build{}, fmt.Errorf("vLLM publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
if h.NVIDIADriver == "" {
|
||||
return Build{}, NotHere{"vLLM runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
pypi := func(v string) Build {
|
||||
return Build{
|
||||
Variant: v,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := func() (Build, error) {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeForCUDA12, h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
|
||||
b := pypi("cu129")
|
||||
b.Requirements = nil
|
||||
b.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
|
||||
Asset: fmt.Sprintf("vllm-%s+cu129-cp38-abi3-manylinux_2_28_%s.whl", version, arch),
|
||||
}
|
||||
b.ExtraIndex = pytorchCU129
|
||||
return b, nil
|
||||
}
|
||||
modern := !versionBefore(version, cuda13Since["vLLM"])
|
||||
switch variant {
|
||||
case "":
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err == nil {
|
||||
return pypi(v), nil
|
||||
}
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) || !modern {
|
||||
return Build{}, err
|
||||
}
|
||||
b, err129 := cu129()
|
||||
if err129 != nil {
|
||||
return Build{}, fmt.Errorf("%v; nor can its CUDA 12.9 build: %v", err, err129)
|
||||
}
|
||||
return b, nil
|
||||
case "cu13":
|
||||
if !modern {
|
||||
return Build{}, fmt.Errorf("vLLM %s is built for CUDA 12; there is no CUDA 13 build of it", version)
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi("cu13"), nil
|
||||
case "cu129":
|
||||
if !modern {
|
||||
// Before 0.26 PyPI's own build is CUDA 12, and is the one given.
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi(v), nil
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) {
|
||||
return Build{}, err
|
||||
}
|
||||
}
|
||||
return cu129()
|
||||
}
|
||||
return Build{}, fmt.Errorf("vLLM has no build %q; the builds are %s", variant, strings.Join(VLLMVariants, ", "))
|
||||
}
|
||||
|
||||
// cudaFits checks a node against a CUDA's driver and GPU requirements.
|
||||
func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f",
|
||||
engine, version, cuda, minCC, cc)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chooseSGLang(version, chosen string, h discovery.Host) (Build, error) {
|
||||
if chosen != "" {
|
||||
return Build{}, fmt.Errorf("SGLang is installed in one build only, not %q", chosen)
|
||||
}
|
||||
variant, err := cudaGate("SGLang", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
return Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// leadingInt is the number a version string starts with: 580 of "580.65.06".
|
||||
func leadingInt(s string) int {
|
||||
n := 0
|
||||
@@ -365,17 +420,30 @@ func leadingInt(s string) int {
|
||||
return n
|
||||
}
|
||||
|
||||
// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
|
||||
// carries a matching cudart archive, so the node needs a driver and nothing
|
||||
// else — no toolkit.
|
||||
const cudaForLlamaCpp = "12.8"
|
||||
// --- llama.cpp ---
|
||||
|
||||
func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them
|
||||
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
|
||||
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
|
||||
// name decides nothing here. The build is tried everywhere, and ldd says
|
||||
// whether this system can load it — in the words of the missing symbol.
|
||||
// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
|
||||
// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
|
||||
// carries as PTX, which a driver older than the CUDA it came from cannot
|
||||
// compile.
|
||||
var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
|
||||
|
||||
// llamaCppCandidates are the llama.cpp builds for a machine.
|
||||
//
|
||||
// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
|
||||
// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
|
||||
// the GPUs it ships as PTX, 525 for those it ships as machine code), and
|
||||
// Vulkan, which runs on any card whose driver has Vulkan — the generations
|
||||
// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
|
||||
// a GPU placement served on the CPU, which is what installing checks for.
|
||||
// The CPU build where there is no NVIDIA GPU.
|
||||
//
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them is
|
||||
// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
|
||||
// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
|
||||
// nothing here. The build is tried everywhere, and ldd says whether this
|
||||
// system can load it — in the words of the missing symbol.
|
||||
func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -383,36 +451,53 @@ func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
|
||||
case "arm64":
|
||||
arch = "arm64"
|
||||
default:
|
||||
return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
|
||||
b := Build{
|
||||
Variant: "cpu",
|
||||
Assets: []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
|
||||
// Everything else the build links is in the archive or in any current
|
||||
// glibc system; OpenMP's runtime is the one a cloud image lacks, and
|
||||
// the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the best
|
||||
// one, down to SSE4.2. Below AVX2 it runs, several times slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
// The CUDA build is published for x64 at this version; on arm64 it is a
|
||||
// newer CUDA that the driver may not support, so an arm64 GPU node runs
|
||||
// the CPU build until someone decides otherwise.
|
||||
if h.NVIDIADriver != "" && arch == "x64" {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
|
||||
b.Variant = "cuda-" + cudaForLlamaCpp
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
b.Assets = []string{
|
||||
fmt.Sprintf("llama-%s-%s", version, name),
|
||||
fmt.Sprintf("cudart-llama-%s-%s", version, name),
|
||||
base := func(variant string, assets ...string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Assets: assets,
|
||||
// Everything else the build links is in the archive or in any
|
||||
// current glibc system; OpenMP's runtime is the one a cloud image
|
||||
// lacks, and the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the
|
||||
// best one, down to SSE4.2. Below AVX2 it runs, several times
|
||||
// slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
return b
|
||||
}
|
||||
return b, nil
|
||||
if h.NVIDIADriver == "" {
|
||||
return pick("llama.cpp", variant, []candidate{
|
||||
{base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
|
||||
}, h)
|
||||
}
|
||||
cuda := func(cv string) Build {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
|
||||
// The release carries the CUDA runtime beside the build, so the node
|
||||
// needs a driver and nothing else — no toolkit.
|
||||
b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
return b
|
||||
}
|
||||
vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
|
||||
vulkan.GPUBackend = "libggml-vulkan.so"
|
||||
vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
|
||||
|
||||
cuda12Driver := 570
|
||||
if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
|
||||
cuda12Driver = minDriverForCUDA12
|
||||
}
|
||||
cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
|
||||
if arch == "x64" {
|
||||
cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
|
||||
}
|
||||
cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
|
||||
return pick("llama.cpp", variant, cands, h)
|
||||
}
|
||||
|
||||
func orUnknown(s string) string {
|
||||
|
||||
@@ -22,7 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it.\n\n" +
|
||||
"cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
|
||||
"gets 0.19.1, the newest built for it, by itself.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
|
||||
@@ -585,6 +585,10 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
|
||||
out.GPUs = append(out.GPUs, inventory.GPUSensor{
|
||||
Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
|
||||
PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
|
||||
UtilPercent: g.GetUtilizationPercent(), MemUtilPercent: g.GetMemoryUtilizationPercent(),
|
||||
MemoryUsedMB: g.GetMemoryUsedMb(), MemoryTotalMB: g.GetMemoryTotalMb(),
|
||||
SMClockMHz: g.GetSmClockMhz(), ECCUncorrected: g.GetEccUncorrected(),
|
||||
Wear: wearFrom(g),
|
||||
})
|
||||
}
|
||||
return out
|
||||
@@ -961,3 +965,22 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
|
||||
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
|
||||
}
|
||||
}
|
||||
|
||||
// wearFrom is a card's wear, when the agent read it.
|
||||
func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
|
||||
if !g.GetHasWear() {
|
||||
return nil
|
||||
}
|
||||
w := &inventory.GPUWear{
|
||||
PCIeReplays: g.GetPcieReplays(), RetiredSBE: g.GetRetiredPagesSbe(), RetiredDBE: g.GetRetiredPagesDbe(),
|
||||
RetirePending: g.GetRetirementPending(), RemappedCorrectable: g.GetRemappedRowsCorrectable(),
|
||||
RemappedUncorrectable: g.GetRemappedRowsUncorrectable(), RemapPending: g.GetRemapPending(),
|
||||
RemapFailure: g.GetRemapFailure(), NVLinkCRC: g.GetNvlinkCrcErrors(), NVLinkReplay: g.GetNvlinkReplayErrors(),
|
||||
NVLinkRecovery: g.GetNvlinkRecoveryErrors(), XIDs: g.GetXidErrors(), LastXID: int(g.GetLastXid()),
|
||||
LastXIDText: g.GetLastXidText(),
|
||||
}
|
||||
if u := g.GetLastXidUnix(); u > 0 {
|
||||
w.LastXIDAt = time.Unix(u, 0)
|
||||
}
|
||||
return w
|
||||
}
|
||||
@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
|
||||
if p.Nodes <= 0 {
|
||||
p.Nodes = 1
|
||||
}
|
||||
// tp on a llama.cpp GPU profile, written before gpus_per_node meant
|
||||
// anything for it, is read as the width it was meant as: N cards of
|
||||
// one node. Refusing it would have dropped a catalog's model on the
|
||||
// upgrade that made the width work. Only both given, disagreeing, is
|
||||
// refused below, as ambiguous.
|
||||
if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
|
||||
if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
|
||||
p.GPUsPerNode, p.TP, p.PP = n, 1, 0
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case p.CPUOnly():
|
||||
// Nothing to divide up: this profile asks for no cards at all.
|
||||
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
|
||||
p.Weight = 1
|
||||
}
|
||||
}
|
||||
// A Hugging Face repository page for a model vLLM or SGLang runs is the
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -335,9 +352,11 @@ func (m Model) validate() error {
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not its files; "+
|
||||
"mirror it and use the version's source ending in /, or give %q for vLLM to fetch itself",
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
@@ -398,8 +417,9 @@ func (m Model) validate() error {
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
m.Name, p.Name, p.Nodes)
|
||||
case p.TP > 1 || p.PP > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, not by tp or pp: "+
|
||||
"write gpus_per_node: %d and leave tp and pp out", m.Name, p.Name, max(p.TP, 1)*max(p.PP, 1))
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
|
||||
"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
|
||||
max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -332,37 +332,56 @@ func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
|
||||
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
|
||||
t.Errorf("placement was not asked for four GPUs: %+v", pl)
|
||||
}
|
||||
// tp is a width llama.cpp has no way to use; refused, saying what to
|
||||
// write instead.
|
||||
// tp, written before gpus_per_node meant anything for llama.cpp, is read
|
||||
// as the same width rather than refused: refusing dropped the model on
|
||||
// the upgrade that made the width work.
|
||||
old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
|
||||
c, err = New([]Model{old})
|
||||
if err != nil {
|
||||
t.Fatalf("tp 4 on llama.cpp: %v", err)
|
||||
}
|
||||
got, _ := c.Resolve("old")
|
||||
if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
|
||||
t.Errorf("tp 4 should place on four GPUs: %+v", p)
|
||||
}
|
||||
// Both given and disagreeing is ambiguous, and refused; so is more than
|
||||
// one node.
|
||||
for _, p := range []Profile{
|
||||
{Name: "tp", Engine: "llamacpp", GPUClass: "t4", TP: 4},
|
||||
{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
|
||||
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
|
||||
} {
|
||||
_, err := New([]Model{model("x", p)})
|
||||
if err == nil {
|
||||
if _, err := New([]Model{model("x", p)}); err == nil {
|
||||
t.Errorf("%s: accepted", p.Name)
|
||||
}
|
||||
if p.TP == 4 && (err == nil || !strings.Contains(err.Error(), "gpus_per_node: 4")) {
|
||||
t.Errorf("tp 4 should point at gpus_per_node: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryPageIsNotASource(t *testing.T) {
|
||||
func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
|
||||
// The field: an AWQ model's page given as a vLLM source, fetched as one
|
||||
// file of HTML. vLLM downloads a repository id itself.
|
||||
for _, src := range []string{
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
|
||||
} {
|
||||
m := model("awq")
|
||||
m.Source = src
|
||||
_, err := New([]Model{m})
|
||||
if err == nil || !strings.Contains(err.Error(), `"Qwen/Qwen2.5-32B-Instruct-AWQ"`) {
|
||||
t.Errorf("%s: %v", src, err)
|
||||
c, err := New([]Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", src, err)
|
||||
}
|
||||
if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
|
||||
t.Errorf("%s became %q", src, got.Source)
|
||||
}
|
||||
}
|
||||
m := model("file")
|
||||
m.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{m}); err != nil {
|
||||
// llama.cpp needs a file, which a page is not.
|
||||
g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
|
||||
g.Source = "https://huggingface.co/Qwen/X-GGUF"
|
||||
if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
|
||||
t.Errorf("a page for llama.cpp: %v", err)
|
||||
}
|
||||
f := model("file")
|
||||
f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{f}); err != nil {
|
||||
t.Errorf("a file under /resolve/ is a source: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -417,3 +417,17 @@ func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
|
||||
|
||||
// EngineChoices are the engine builds chosen per node, or nil.
|
||||
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
|
||||
|
||||
// ModelOwners maps each model to the part of the tenancy tree its capacity
|
||||
// counts against; a model owned by nobody is not in it.
|
||||
func (c *Controller) ModelOwners() map[string]string {
|
||||
out := map[string]string{}
|
||||
if cat := c.Catalog(); cat != nil {
|
||||
for _, m := range cat.Models() {
|
||||
if m.Owner != "" {
|
||||
out[m.Name] = m.Owner
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -73,6 +73,18 @@ type Collector struct {
|
||||
gpuMemoryTotal *prometheus.Desc
|
||||
gpuMemoryUsed *prometheus.Desc
|
||||
gpuUtil *prometheus.Desc
|
||||
gpuAssigned *prometheus.Desc
|
||||
gpuPCIeReplay *prometheus.Desc
|
||||
gpuRetired *prometheus.Desc
|
||||
gpuRetirePend *prometheus.Desc
|
||||
gpuRemapped *prometheus.Desc
|
||||
gpuRemapPend *prometheus.Desc
|
||||
gpuRemapFail *prometheus.Desc
|
||||
gpuNVLinkErr *prometheus.Desc
|
||||
gpuXID *prometheus.Desc
|
||||
gpuLastXID *prometheus.Desc
|
||||
gpuMemUtil *prometheus.Desc
|
||||
gpuSMClock *prometheus.Desc
|
||||
gpuTemp *prometheus.Desc
|
||||
gpuMemTemp *prometheus.Desc
|
||||
gpuPower *prometheus.Desc
|
||||
@@ -149,6 +161,33 @@ func New(src Source) *Collector {
|
||||
"How busy a GPU was over the last sample period. Absent when the driver does not "+
|
||||
"report it, which is not the same as idle. Memory used says a model is loaded; "+
|
||||
"this says whether it is doing anything.", gpu, nil),
|
||||
gpuPCIeReplay: prometheus.NewDesc("llmgw_gpu_pcie_replays",
|
||||
"PCIe link retries on a GPU's link since the driver loaded; one that keeps climbing is a riser, slot or cable.", gpu, nil),
|
||||
gpuRetired: prometheus.NewDesc("llmgw_gpu_retired_pages",
|
||||
"Memory pages retired, by cause (sbe, dbe), on GPUs before Ampere.", append(gpu, "cause"), nil),
|
||||
gpuRetirePend: prometheus.NewDesc("llmgw_gpu_retirement_pending",
|
||||
"1 when retired pages wait for a GPU reset to take effect.", gpu, nil),
|
||||
gpuRemapped: prometheus.NewDesc("llmgw_gpu_remapped_rows",
|
||||
"Memory rows remapped, by kind (correctable, uncorrectable), on Ampere and later.", append(gpu, "kind"), nil),
|
||||
gpuRemapPend: prometheus.NewDesc("llmgw_gpu_remap_pending",
|
||||
"1 when remapped rows wait for a GPU reset to take effect.", gpu, nil),
|
||||
gpuRemapFail: prometheus.NewDesc("llmgw_gpu_remap_failure",
|
||||
"1 when a GPU has run out of spare memory rows: it needs replacing.", gpu, nil),
|
||||
gpuNVLinkErr: prometheus.NewDesc("llmgw_gpu_nvlink_errors",
|
||||
"NVLink errors summed over a GPU's links, by kind (crc, replay, recovery).", append(gpu, "kind"), nil),
|
||||
gpuXID: prometheus.NewDesc("llmgw_gpu_xid_errors",
|
||||
"The driver's XID error events for a GPU since its node's agent started.", gpu, nil),
|
||||
gpuLastXID: prometheus.NewDesc("llmgw_gpu_last_xid",
|
||||
"The code of a GPU's last XID error since its agent started; absent when there was none.", gpu, nil),
|
||||
gpuAssigned: prometheus.NewDesc("llmgw_gpu_assigned",
|
||||
"1 for each GPU a replica holds: which model, which replica, and whose quota it counts against. "+
|
||||
"Joined on node and gpu with the other llmgw_gpu_ series, it says what each model's cards are doing.",
|
||||
[]string{"node", "gpu", "model", "replica", "owner"}, nil),
|
||||
gpuMemUtil: prometheus.NewDesc("llmgw_gpu_memory_utilization_percent",
|
||||
"How much of the sample period a GPU's memory was being read or written: "+
|
||||
"decode is bound by it, so a card busy here and idle above is waiting on memory.", gpu, nil),
|
||||
gpuSMClock: prometheus.NewDesc("llmgw_gpu_sm_clock_mhz",
|
||||
"A GPU's streaming multiprocessor clock now; one held low under load is the card being throttled.", gpu, nil),
|
||||
gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
|
||||
"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
|
||||
gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
|
||||
@@ -271,7 +310,9 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
|
||||
c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
|
||||
c.gpuNVLJoined, c.imexUp,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuPCIeReplay, c.gpuRetired, c.gpuRetirePend, c.gpuRemapped, c.gpuRemapPend, c.gpuRemapFail,
|
||||
c.gpuNVLinkErr, c.gpuXID, c.gpuLastXID,
|
||||
c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
|
||||
c.gpuNVLink, c.gpuECC,
|
||||
c.replicaState, c.replicaReady, c.replicaIdle,
|
||||
@@ -287,6 +328,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
now := time.Now()
|
||||
c.collectQuotas(ch)
|
||||
c.collectAssignments(ch)
|
||||
|
||||
loads := map[string]*inventory.Load{}
|
||||
for _, n := range c.src.Nodes() {
|
||||
@@ -388,20 +430,28 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
byClass[class]++
|
||||
labels := []string{n.ID, strconv.Itoa(g.Index), class}
|
||||
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
|
||||
float64(g.MemoryTotalMB)*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
|
||||
float64(g.MemoryUsedMB)*1024*1024, labels...)
|
||||
if g.UtilizationPercent > 0 {
|
||||
// Only when reported. A driver too old to give it would
|
||||
// otherwise show every card at a flat zero, which reads as an
|
||||
// idle fleet rather than a missing measurement.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
|
||||
float64(g.UtilizationPercent), labels...)
|
||||
s, fresh := sensorFor(n.Load, g.Index, now)
|
||||
if fresh && s.Busy() {
|
||||
// What the card is doing now, sampled every heartbeat.
|
||||
c.gpuBusy(ch, s, labels)
|
||||
} else {
|
||||
// An agent too old to sample it: what it read at
|
||||
// registration, which is all there is.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
|
||||
float64(g.MemoryTotalMB)*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
|
||||
float64(g.MemoryUsedMB)*1024*1024, labels...)
|
||||
if g.UtilizationPercent > 0 {
|
||||
// Only when reported. A driver too old to give it would
|
||||
// otherwise show every card at a flat zero, which reads as
|
||||
// an idle fleet rather than a missing measurement.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
|
||||
float64(g.UtilizationPercent), labels...)
|
||||
}
|
||||
}
|
||||
// The sensors' fresh reading when there is one; the temperature
|
||||
// read at registration otherwise, which is all an older agent sends.
|
||||
if s, ok := sensorFor(n.Load, g.Index, now); ok {
|
||||
if fresh {
|
||||
c.gpuSensor(ch, s, labels)
|
||||
} else if g.Health.TemperatureC > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
|
||||
@@ -414,8 +464,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuHealthy, prometheus.GaugeValue, healthy, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuNVLink, prometheus.GaugeValue,
|
||||
float64(len(g.NVLinkPeers)), labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue,
|
||||
float64(g.Health.ECCErrors), labels...)
|
||||
ecc := float64(g.Health.ECCErrors)
|
||||
if fresh && s.Busy() && s.ECCUncorrected >= 0 {
|
||||
ecc = s.ECCUncorrected
|
||||
}
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue, ecc, labels...)
|
||||
}
|
||||
for class, n2 := range byClass {
|
||||
ch <- prometheus.MustNewConstMetric(c.nodeGPUs, prometheus.GaugeValue,
|
||||
@@ -521,6 +574,12 @@ func b2f(b bool) float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
// OwnerSource says whose quota each model counts against, for labelling
|
||||
// what its replicas hold. Asked for by type assertion.
|
||||
type OwnerSource interface {
|
||||
ModelOwners() map[string]string
|
||||
}
|
||||
|
||||
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
|
||||
// assertion, like HealthSource.
|
||||
type QuotaSource interface {
|
||||
@@ -569,6 +628,23 @@ func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor
|
||||
return inventory.GPUSensor{}, false
|
||||
}
|
||||
|
||||
// gpuBusy is how busy a card is, from a sample that carries it.
|
||||
func (c *Collector) gpuBusy(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue, s.MemoryTotalMB*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue, s.MemoryUsedMB*1024*1024, labels...)
|
||||
// Zero is a real reading here — an idle card — and only -1, the driver
|
||||
// not saying, leaves the series out.
|
||||
if s.UtilPercent >= 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue, s.UtilPercent, labels...)
|
||||
}
|
||||
if s.MemUtilPercent >= 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemUtil, prometheus.GaugeValue, s.MemUtilPercent, labels...)
|
||||
}
|
||||
if s.SMClockMHz > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuSMClock, prometheus.GaugeValue, s.SMClockMHz, labels...)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
|
||||
for _, v := range []struct {
|
||||
d *prometheus.Desc
|
||||
@@ -581,4 +657,61 @@ func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor
|
||||
for _, reason := range s.Throttle {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
|
||||
}
|
||||
if w := s.Wear; w != nil {
|
||||
c.gpuWear(ch, w, labels)
|
||||
}
|
||||
}
|
||||
|
||||
// gpuWear is what wears a card out. A count the card does not keep (-1) is
|
||||
// left out, not published as zero.
|
||||
func (c *Collector) gpuWear(ch chan<- prometheus.Metric, w *inventory.GPUWear, labels []string) {
|
||||
gauge := func(d *prometheus.Desc, v float64, extra ...string) {
|
||||
ch <- prometheus.MustNewConstMetric(d, prometheus.GaugeValue, v, append(append([]string{}, labels...), extra...)...)
|
||||
}
|
||||
flag := func(b bool) float64 {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
if w.PCIeReplays >= 0 {
|
||||
gauge(c.gpuPCIeReplay, float64(w.PCIeReplays))
|
||||
}
|
||||
if w.RetiredSBE >= 0 || w.RetiredDBE >= 0 {
|
||||
gauge(c.gpuRetired, float64(max(w.RetiredSBE, 0)), "sbe")
|
||||
gauge(c.gpuRetired, float64(max(w.RetiredDBE, 0)), "dbe")
|
||||
gauge(c.gpuRetirePend, flag(w.RetirePending))
|
||||
}
|
||||
if w.RemappedCorrectable >= 0 || w.RemappedUncorrectable >= 0 {
|
||||
gauge(c.gpuRemapped, float64(max(w.RemappedCorrectable, 0)), "correctable")
|
||||
gauge(c.gpuRemapped, float64(max(w.RemappedUncorrectable, 0)), "uncorrectable")
|
||||
gauge(c.gpuRemapPend, flag(w.RemapPending))
|
||||
gauge(c.gpuRemapFail, flag(w.RemapFailure))
|
||||
}
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkCRC), "crc")
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkReplay), "replay")
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkRecovery), "recovery")
|
||||
gauge(c.gpuXID, float64(w.XIDs))
|
||||
if w.XIDs > 0 {
|
||||
gauge(c.gpuLastXID, float64(w.LastXID))
|
||||
}
|
||||
}
|
||||
|
||||
// collectAssignments says which replica holds each GPU.
|
||||
func (c *Collector) collectAssignments(ch chan<- prometheus.Metric) {
|
||||
var owners map[string]string
|
||||
if o, ok := c.src.(OwnerSource); ok {
|
||||
owners = o.ModelOwners()
|
||||
}
|
||||
for _, r := range c.src.Replicas() {
|
||||
if !r.State.Counts() {
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
for _, g := range sh.GPUIndices {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuAssigned, prometheus.GaugeValue, 1,
|
||||
sh.NodeID, strconv.Itoa(g), r.Model, r.ID, owners[r.Model])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
@@ -39,3 +40,90 @@ func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHowBusyAGPUIsComesFromTheLatestSampleNotRegistration(t *testing.T) {
|
||||
// Discovery read 87% and 70 GB used when the agent came up; the card has
|
||||
// been idle since. The sample says so, zero included.
|
||||
reg := discovery.GPU{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
|
||||
MemoryUsedMB: 70000, UtilizationPercent: 87}
|
||||
n := inventory.Node{ID: "g1", State: inventory.Ready, GPUs: []discovery.GPU{reg},
|
||||
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 35,
|
||||
UtilPercent: 0, MemUtilPercent: 0, MemoryUsedMB: 512, MemoryTotalMB: 81559,
|
||||
SMClockMHz: 345, ECCUncorrected: 2}}}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
|
||||
"llmgw_gpu_memory_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
|
||||
"llmgw_gpu_memory_used_bytes class=h100-80g gpu=0 node=g1 -> 536870912\n",
|
||||
"llmgw_gpu_sm_clock_mhz class=h100-80g gpu=0 node=g1 -> 345\n",
|
||||
"llmgw_gpu_ecc_errors class=h100-80g gpu=0 node=g1 -> 2\n",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s in:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "-> 87\n") {
|
||||
t.Error("the utilization read at registration was published over the sample's")
|
||||
}
|
||||
|
||||
// An agent too old to sample it: registration's figures, and no zero
|
||||
// standing in for a utilization the driver did not give.
|
||||
n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35}}
|
||||
out = gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
if !strings.Contains(out, "llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 87") {
|
||||
t.Errorf("an older agent's registration figure was dropped:\n%s", out)
|
||||
}
|
||||
n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35, UtilPercent: -1, MemUtilPercent: -1, MemoryTotalMB: 81559}}
|
||||
if out := gather(t, fakeSource{nodes: []inventory.Node{n}}); strings.Contains(out, "llmgw_gpu_utilization_percent ") {
|
||||
t.Errorf("a utilization the driver did not report was published:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// ownedSource adds owners to fakeSource.
|
||||
type ownedSource struct{ fakeSource }
|
||||
|
||||
func (ownedSource) ModelOwners() map[string]string { return map[string]string{"chat": "acme/ml"} }
|
||||
|
||||
func TestEachGPUAReplicaHoldsSaysWhoseItIs(t *testing.T) {
|
||||
src := ownedSource{fakeSource{replicas: []reconciler.Replica{
|
||||
{ID: "chat-1", Model: "chat", State: reconciler.Ready, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{0, 1}}}},
|
||||
{ID: "old-1", Model: "old", State: reconciler.Stopped, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}},
|
||||
}}}
|
||||
out := gather(t, src)
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_assigned gpu=0 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
|
||||
"llmgw_gpu_assigned gpu=1 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s in:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "replica=old-1") {
|
||||
t.Error("a stopped replica's GPU was counted as held")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGPUsWearIsPublishedAndWhatItDoesNotKeepIsLeftOut(t *testing.T) {
|
||||
n := inventory.Node{ID: "g1", State: inventory.Ready,
|
||||
GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559}},
|
||||
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 40, MemoryTotalMB: 81559,
|
||||
Wear: &inventory.GPUWear{PCIeReplays: 12, RetiredSBE: -1, RetiredDBE: -1, RemappedCorrectable: 2,
|
||||
RemappedUncorrectable: 1, RemapPending: true, NVLinkCRC: 7, XIDs: 3, LastXID: 79}}}}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_pcie_replays class=h100-80g gpu=0 node=g1 -> 12\n",
|
||||
"llmgw_gpu_remapped_rows class=h100-80g gpu=0 kind=uncorrectable node=g1 -> 1\n",
|
||||
"llmgw_gpu_remap_pending class=h100-80g gpu=0 node=g1 -> 1\n",
|
||||
"llmgw_gpu_nvlink_errors class=h100-80g gpu=0 kind=crc node=g1 -> 7\n",
|
||||
"llmgw_gpu_xid_errors class=h100-80g gpu=0 node=g1 -> 3\n",
|
||||
"llmgw_gpu_last_xid class=h100-80g gpu=0 node=g1 -> 79\n",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s", want)
|
||||
}
|
||||
}
|
||||
// An H100 keeps no retired pages; none are published.
|
||||
if strings.Contains(out, "llmgw_gpu_retired_pages") {
|
||||
t.Errorf("a count the card does not keep was published:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@
|
||||
package inventory
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -319,8 +320,74 @@ type GPUSensor struct {
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerLimitW float64 `json:"power_limit_w,omitempty"`
|
||||
Throttle []string `json:"throttle,omitempty"`
|
||||
// How busy the card is, as of this sample; -1 is not reported. Zero
|
||||
// MemoryTotalMB is an agent too old to send these, whose figures are
|
||||
// the ones it read at registration.
|
||||
UtilPercent float64 `json:"util_percent,omitempty"`
|
||||
MemUtilPercent float64 `json:"mem_util_percent,omitempty"`
|
||||
MemoryUsedMB float64 `json:"memory_used_mb,omitempty"`
|
||||
MemoryTotalMB float64 `json:"memory_total_mb,omitempty"`
|
||||
SMClockMHz float64 `json:"sm_clock_mhz,omitempty"`
|
||||
ECCUncorrected float64 `json:"ecc_uncorrected,omitempty"`
|
||||
// Wear is what wears the card out, read every minute; nil from an agent
|
||||
// that does not read it.
|
||||
Wear *GPUWear `json:"wear,omitempty"`
|
||||
}
|
||||
|
||||
// GPUWear is a card's wear and failures. Counts of -1 are not reported for
|
||||
// it: a card before Ampere has no remapped rows, one after no retired pages.
|
||||
type GPUWear struct {
|
||||
PCIeReplays int64 `json:"pcie_replays"`
|
||||
RetiredSBE int64 `json:"retired_pages_sbe"`
|
||||
RetiredDBE int64 `json:"retired_pages_dbe"`
|
||||
RetirePending bool `json:"retire_pending,omitempty"`
|
||||
RemappedCorrectable int64 `json:"remapped_rows_correctable"`
|
||||
RemappedUncorrectable int64 `json:"remapped_rows_uncorrectable"`
|
||||
RemapPending bool `json:"remap_pending,omitempty"`
|
||||
RemapFailure bool `json:"remap_failure,omitempty"`
|
||||
NVLinkCRC int64 `json:"nvlink_crc_errors"`
|
||||
NVLinkReplay int64 `json:"nvlink_replay_errors"`
|
||||
NVLinkRecovery int64 `json:"nvlink_recovery_errors"`
|
||||
XIDs int64 `json:"xid_errors"`
|
||||
LastXID int `json:"last_xid,omitempty"`
|
||||
LastXIDText string `json:"last_xid_text,omitempty"`
|
||||
LastXIDAt time.Time `json:"last_xid_at,omitempty"`
|
||||
}
|
||||
|
||||
// Problems says, in a few words each, what is wrong with the card; empty
|
||||
// when nothing is.
|
||||
func (w *GPUWear) Problems() []string {
|
||||
if w == nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
if w.RemapFailure {
|
||||
out = append(out, "out of spare memory rows: replace the card")
|
||||
}
|
||||
if w.RemapPending {
|
||||
out = append(out, "memory rows remapped, waiting for a reset")
|
||||
}
|
||||
if w.RetirePending {
|
||||
out = append(out, "memory pages waiting for a reset to be retired")
|
||||
}
|
||||
if w.RemappedUncorrectable > 0 {
|
||||
out = append(out, fmt.Sprintf("%d rows remapped for uncorrectable errors", w.RemappedUncorrectable))
|
||||
}
|
||||
if w.RetiredDBE > 0 {
|
||||
out = append(out, fmt.Sprintf("%d pages retired for double-bit errors", w.RetiredDBE))
|
||||
}
|
||||
if w.XIDs > 0 {
|
||||
out = append(out, fmt.Sprintf("XID %d (%s)", w.LastXID, w.LastXIDText))
|
||||
}
|
||||
if w.NVLinkCRC > 0 || w.NVLinkReplay > 0 || w.NVLinkRecovery > 0 {
|
||||
out = append(out, fmt.Sprintf("NVLink errors: %d CRC, %d replay, %d recovery", w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Busy reports whether the sample carries how busy the card is.
|
||||
func (s GPUSensor) Busy() bool { return s.MemoryTotalMB > 0 }
|
||||
|
||||
// ShardLoad is one engine's use: cores busy, and resident memory.
|
||||
type ShardLoad struct {
|
||||
CPUCores float64
|
||||
|
||||
@@ -581,7 +581,16 @@ func Evaluate(f Facts) []Finding {
|
||||
// the same decision the agent makes, so the report and the node agree.
|
||||
func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
v := f.VLLMVersion
|
||||
b, err := provision.Recipes["vllm"].Choose(v, "", f.Host)
|
||||
builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
|
||||
var b provision.Build
|
||||
if err == nil {
|
||||
// The first the agent would try; it moves to the next by itself if
|
||||
// that one does not work here.
|
||||
b = builds[0]
|
||||
if b.Version != "" {
|
||||
v = b.Version
|
||||
}
|
||||
}
|
||||
var nh provision.NotHere
|
||||
switch {
|
||||
case errors.As(err, &nh):
|
||||
@@ -592,8 +601,7 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
// llama.cpp. Refusing it here kept older GPU generations out of a
|
||||
// cluster for an engine nobody meant to run on them.
|
||||
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
|
||||
"give this node a vLLM it can run with engine_overrides in the server's configuration, "+
|
||||
"or add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
"add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
return
|
||||
case f.VLLMInstalled:
|
||||
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
|
||||
@@ -605,8 +613,12 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
"free space there, or point engines_dir in agent.yaml at a larger disk")
|
||||
return
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
|
||||
note := ""
|
||||
if b.Note != "" {
|
||||
note = "; " + b.Note
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
|
||||
}
|
||||
|
||||
func rank(l Level) int {
|
||||
|
||||
@@ -154,11 +154,17 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
|
||||
// What CUDA 13 cannot run, its CUDA 12.9 build of the same version can.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
|
||||
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
// A V100 has no build of vLLM 0.20 or later; it gets the newest
|
||||
// that has Volta kernels, and doctor says which.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
|
||||
{"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
|
||||
VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
|
||||
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
|
||||
@@ -203,6 +203,9 @@ func (r *Router) Handler() http.Handler {
|
||||
}
|
||||
pages.RegisterAdminQuotas(mux, r.store, r.acct)
|
||||
// And the report of what that tree actually used.
|
||||
if r.history != nil {
|
||||
pages.SetHistory(r.history)
|
||||
}
|
||||
pages.RegisterAdminUsage(mux, r.store, r.acct)
|
||||
}
|
||||
|
||||
|
||||
@@ -64,6 +64,9 @@ type AccountPages struct {
|
||||
// hasQuotas is set when /admin/quotas is registered.
|
||||
hasQuotas bool
|
||||
// capacity sets capacity quotas, from the quotas page; nil leaves them off.
|
||||
// history is the metrics store, for what only it knows: the GPU time
|
||||
// each owner held. Nil without one.
|
||||
history History
|
||||
capacity CapacityAdmin
|
||||
// personalOff refuses personal keys; see SetPersonalOff.
|
||||
personalOff bool
|
||||
@@ -651,3 +654,6 @@ func errorRef() string {
|
||||
_, _ = rand.Read(b[:])
|
||||
return "err-" + hex.EncodeToString(b[:])
|
||||
}
|
||||
|
||||
// SetHistory gives the pages the metrics store.
|
||||
func (p *AccountPages) SetHistory(h History) { p.history = h }
|
||||
@@ -52,7 +52,7 @@ func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
|
||||
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
|
||||
t.Error("a node with no GPU was offered a vLLM build")
|
||||
}
|
||||
if !strings.Contains(body, "CUDA 12.9 installing") {
|
||||
if !strings.Contains(body, "0.30.0 CUDA 12.9 installing") {
|
||||
t.Error("what is installed is not shown beside the choice")
|
||||
}
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
@@ -80,6 +81,12 @@ type adminUsageData struct {
|
||||
// chart along the bottom reads as "nobody spent anything".
|
||||
Priced bool
|
||||
|
||||
// GPUHeld and GPUHours are the cards each owner's models held over the
|
||||
// window, from the metrics store; empty without one, or with no owned
|
||||
// model on a GPU.
|
||||
GPUHeld TimeSeries
|
||||
GPUHours []gpuHoursRow
|
||||
|
||||
Error string
|
||||
Version string
|
||||
Now string
|
||||
@@ -135,9 +142,46 @@ func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
|
||||
d.Requests, d.Tokens, d.Spend, d.Rows = usageCharts(rows)
|
||||
d.GPUHeld, d.GPUHours = p.gpuHours(r.Context(), win)
|
||||
render(w, "admin_usage", d)
|
||||
}
|
||||
|
||||
// gpuHoursRow is the GPU time one owner's models held over the window.
|
||||
type gpuHoursRow struct {
|
||||
Owner string
|
||||
Hours float64
|
||||
// Now is how many cards they hold at the end of the window.
|
||||
Now float64
|
||||
}
|
||||
|
||||
// gpuHours is the cards held by each owner's models over the window, drawn
|
||||
// and summed into GPU-hours. From the metrics store, because what a model
|
||||
// holds is placement's, not the request log's: a model nobody asks still
|
||||
// holds its cards. As far back as the store keeps history, which for a
|
||||
// default Prometheus is fifteen days.
|
||||
func (p *AccountPages) gpuHours(ctx context.Context, win usageWindow) (TimeSeries, []gpuHoursRow) {
|
||||
if p.history == nil {
|
||||
return TimeSeries{}, nil
|
||||
}
|
||||
series, err := p.history.RangeQuery(ctx, `sum by (owner) (llmgw_gpu_assigned)`, "owner", win.Span, win.Width)
|
||||
if err != nil || len(series) == 0 {
|
||||
return TimeSeries{}, nil
|
||||
}
|
||||
var rows []gpuHoursRow
|
||||
for i := range series {
|
||||
if series[i].Name == "" {
|
||||
series[i].Name = "no owner"
|
||||
}
|
||||
row := gpuHoursRow{Owner: series[i].Name, Now: series[i].Last()}
|
||||
// Each point is the cards held at that instant, standing for the
|
||||
// step that follows it.
|
||||
row.Hours = series[i].Sum() * win.Width.Hours()
|
||||
rows = append(rows, row)
|
||||
}
|
||||
sort.Slice(rows, func(i, j int) bool { return rows[i].Hours > rows[j].Hours })
|
||||
return LineChart(series, "", 6), rows
|
||||
}
|
||||
|
||||
// seriesOf turns each group's buckets into a line of one measure.
|
||||
//
|
||||
// One pass per measure rather than one query per measure: the buckets are
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -162,3 +163,32 @@ func TestAPersonWithNoAccountIsNamedRatherThanBlank(t *testing.T) {
|
||||
t.Errorf("unowned traffic was left unlabelled; got %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheUsagePageCountsTheGPUTimeEachOwnerHeld(t *testing.T) {
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
// Over a day in 20-minute steps: team:research held 8 cards throughout,
|
||||
// nobody's model 2 for the last three steps.
|
||||
res, none := promq.Series{Name: "team:research"}, promq.Series{Name: ""}
|
||||
for i := range 72 {
|
||||
res.Points = append(res.Points, promq.Point{Value: 8})
|
||||
if i >= 69 {
|
||||
none.Points = append(none.Points, promq.Point{Value: 2})
|
||||
}
|
||||
}
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{`sum by (owner) (llmgw_gpu_assigned)`: {res, none}}}
|
||||
p.SetHistory(h)
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsage(mux, &fakeScopeSeries{}, nil)
|
||||
body := mirrorGet(t, mux, "/admin/usage?window=24h").Body.String()
|
||||
for _, want := range []string{"GPU time, by owner", "<td>team:research</td><td class=\"num\">192.0</td>",
|
||||
"<td>no owner</td><td class=\"num\">2.0</td>"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// Without a metrics store, nothing is said about GPU time.
|
||||
if body := mirrorGet(t, usagePage(t, &fakeScopeSeries{}, nil), "/admin/usage").Body.String(); strings.Contains(body, "GPU time") {
|
||||
t.Error("GPU time shown with no metrics store to read it from")
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,13 @@ type DashboardData struct {
|
||||
CPU cpuCapacity
|
||||
// Menu is the shared top menu; Nav below is what the cluster page adds.
|
||||
Menu Nav
|
||||
// GPUMap is every card coloured by one figure, and IdleGPUs the replicas
|
||||
// holding cards that are doing nothing; both from the last heartbeats.
|
||||
GPUMap *gpuMap
|
||||
IdleGPUs []idleHolding
|
||||
// Efficiency is what each model gets from its cards, from the metrics
|
||||
// store; nil without one.
|
||||
Efficiency []modelEfficiency
|
||||
// Notice and Problem are what the last change made from this page did,
|
||||
// carried over the redirect back to it.
|
||||
Notice, Problem string
|
||||
@@ -198,9 +205,17 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
d.AnyFabrics = d.AnyFabrics || len(n.Fabrics) > 0
|
||||
}
|
||||
d.NodePrefix = sharedPrefix(ids)
|
||||
if d.AnyGPUs {
|
||||
d.GPUMap = buildGPUMap(d.Nodes, d.Replicas, r.URL.Query().Get("gpumap"), d.NodePrefix)
|
||||
d.IdleGPUs = idleHoldings(d.Nodes, d.Replicas, d.NodePrefix)
|
||||
}
|
||||
for _, m := range d.Models {
|
||||
d.AnyFallbacks = d.AnyFallbacks || len(m.Fallbacks) > 0
|
||||
}
|
||||
if d.AnyGPUs {
|
||||
d.Fleet.LoadGPU(r.Context(), nav.History, nav.Window)
|
||||
d.Efficiency = loadEfficiency(r.Context(), nav.History)
|
||||
}
|
||||
d.Summary = src.Summary()
|
||||
if fs, ok := src.(FabricSource); ok {
|
||||
r := fs.Fabrics()
|
||||
|
||||
@@ -24,7 +24,10 @@ type FleetCharts struct {
|
||||
Queue TimeSeries
|
||||
KV TimeSeries
|
||||
Ready TimeSeries
|
||||
Window string
|
||||
// GPUBusy and GPUMemory are the fleet's cards by class, for a fleet that
|
||||
// has them.
|
||||
GPUBusy, GPUMemory TimeSeries
|
||||
Window string
|
||||
// Err is set when the metrics store could not be read. Shown as a line of
|
||||
// text where the charts would be: an empty panel reads as a healthy cluster
|
||||
// with nothing happening, which is the opposite of not knowing.
|
||||
@@ -90,3 +93,25 @@ func LoadFleetCharts(ctx context.Context, h History, window time.Duration,
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// LoadGPU adds the fleet's GPUs, by class: how busy on average, and how full
|
||||
// their memory. Asked only of a fleet with GPUs. A failure leaves these two
|
||||
// empty and the rest of the page as it was; they are not what the other
|
||||
// panels' all-or-nothing is about.
|
||||
func (f *FleetCharts) LoadGPU(ctx context.Context, h History, window time.Duration) {
|
||||
if f == nil || h == nil || f.Err != "" {
|
||||
return
|
||||
}
|
||||
if window <= 0 {
|
||||
window = time.Hour
|
||||
}
|
||||
step := window / 60
|
||||
if s, err := h.RangeQuery(ctx, `avg by (class) (llmgw_gpu_utilization_percent) / 100`, "class", window, step); err == nil {
|
||||
f.GPUBusy = LineChart(s, "%", 6)
|
||||
}
|
||||
if s, err := h.RangeQuery(ctx,
|
||||
`sum by (class) (llmgw_gpu_memory_used_bytes) / sum by (class) (llmgw_gpu_memory_total_bytes)`,
|
||||
"class", window, step); err == nil {
|
||||
f.GPUMemory = LineChart(s, "%", 6)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// modelEfficiency is what a model gets out of the GPUs it holds, over the
|
||||
// last quarter hour.
|
||||
type modelEfficiency struct {
|
||||
Model string
|
||||
// TokensPerSec is output tokens a second; GPUs the cards it holds now.
|
||||
TokensPerSec float64
|
||||
GPUs float64
|
||||
// Watts is what those cards draw; Util their average utilisation.
|
||||
Watts, Util float64
|
||||
// PerGPU and PerWh are the two ways of saying how much work the cards
|
||||
// do: output tokens a second per card, and per watt-hour drawn.
|
||||
PerGPU, PerWh float64
|
||||
}
|
||||
|
||||
// The queries behind the table. Each model's cards are found by joining the
|
||||
// GPU series to llmgw_gpu_assigned on the card, which carries the model.
|
||||
var efficiencyQueries = []struct {
|
||||
expr string
|
||||
into func(*modelEfficiency, float64)
|
||||
}{
|
||||
{`sum by (model) (rate(llmgw_tokens_total{direction="completion"}[15m]))`,
|
||||
func(e *modelEfficiency, v float64) { e.TokensPerSec = v }},
|
||||
{`count by (model) (llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.GPUs = v }},
|
||||
{`sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.Watts = v }},
|
||||
{`avg by (model) (llmgw_gpu_utilization_percent * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.Util = v }},
|
||||
}
|
||||
|
||||
// loadEfficiency is the table, best per card first; nil when the metrics
|
||||
// store cannot be read or no model holds a GPU.
|
||||
func loadEfficiency(ctx context.Context, h History) []modelEfficiency {
|
||||
if h == nil {
|
||||
return nil
|
||||
}
|
||||
byModel := map[string]*modelEfficiency{}
|
||||
for _, q := range efficiencyQueries {
|
||||
// The last of a short range: what History offers, and the one point
|
||||
// the table needs.
|
||||
series, err := h.RangeQuery(ctx, q.expr, "model", 10*time.Minute, 5*time.Minute)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
for _, s := range series {
|
||||
if len(s.Points) == 0 {
|
||||
continue
|
||||
}
|
||||
e := byModel[s.Name]
|
||||
if e == nil {
|
||||
e = &modelEfficiency{Model: s.Name}
|
||||
byModel[s.Name] = e
|
||||
}
|
||||
q.into(e, s.Last())
|
||||
}
|
||||
}
|
||||
var out []modelEfficiency
|
||||
for _, e := range byModel {
|
||||
if e.GPUs == 0 {
|
||||
continue // a model on the CPU; this table is about cards
|
||||
}
|
||||
e.PerGPU = e.TokensPerSec / e.GPUs
|
||||
if e.Watts > 0 {
|
||||
e.PerWh = e.TokensPerSec * 3600 / e.Watts
|
||||
}
|
||||
out = append(out, *e)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].PerGPU != out[j].PerGPU {
|
||||
return out[i].PerGPU > out[j].PerGPU
|
||||
}
|
||||
return out[i].Model < out[j].Model
|
||||
})
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"html/template"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// The GPU map is every card in the fleet, a square each, a row per node,
|
||||
// coloured by one figure: how busy, how hot, or how full its memory is. A
|
||||
// hundred cards in a table are a hundred rows to read; on the map the one
|
||||
// that is not like the others is seen before anything is read.
|
||||
|
||||
// gpuMapModes are what the map can be coloured by.
|
||||
var gpuMapModes = []struct{ Key, Label string }{
|
||||
{"util", "busy"}, {"memory", "memory"}, {"temp", "temperature"},
|
||||
}
|
||||
|
||||
type gpuMap struct {
|
||||
By string
|
||||
Modes []struct{ Key, Label string }
|
||||
Rows []gpuMapRow
|
||||
// Legend says what the colours mean in this mode.
|
||||
Legend string
|
||||
}
|
||||
|
||||
type gpuMapRow struct {
|
||||
Node string
|
||||
Cells []gpuCell
|
||||
}
|
||||
|
||||
type gpuCell struct {
|
||||
Index int
|
||||
// Model and Replica hold the card; empty is a free card.
|
||||
Model, Replica string
|
||||
// Style is the cell's colour, built here from numbers alone, which is
|
||||
// what makes it safe to put in a style attribute; Title says everything
|
||||
// known about the card.
|
||||
Style template.CSS
|
||||
Title string
|
||||
// Label is the figure written in the cell.
|
||||
Label string
|
||||
// Faulty is a card whose wear needs seeing to.
|
||||
Faulty bool
|
||||
}
|
||||
|
||||
// idleHolding is a replica holding GPUs that are doing nothing.
|
||||
type idleHolding struct {
|
||||
Model, Replica, Node string
|
||||
GPUs []int
|
||||
// MaxUtil is the busiest of its cards, now.
|
||||
MaxUtil float64
|
||||
// Idle is how long since a request was routed to it, or "" when not
|
||||
// known.
|
||||
Idle string
|
||||
}
|
||||
|
||||
// buildGPUMap draws the map from what the nodes last reported and where
|
||||
// replicas are. Nil for a fleet without GPUs.
|
||||
func buildGPUMap(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, by, prefix string) *gpuMap {
|
||||
held := heldBy(reps)
|
||||
m := &gpuMap{By: by, Modes: gpuMapModes}
|
||||
if m.By == "" {
|
||||
m.By = "util"
|
||||
}
|
||||
switch m.By {
|
||||
case "temp":
|
||||
m.Legend = "Blue is cool, red is 90 °C and over."
|
||||
case "memory":
|
||||
m.Legend = "Pale is empty, dark is full. Engines reserve most of a card's memory when they start."
|
||||
default:
|
||||
m.By = "util"
|
||||
m.Legend = "Pale is idle, dark green is fully busy. A dashed border is a card no replica holds."
|
||||
}
|
||||
for _, n := range nodes {
|
||||
if n.GPUTotal == 0 {
|
||||
continue
|
||||
}
|
||||
row := gpuMapRow{Node: strings.TrimPrefix(n.ID, prefix)}
|
||||
for i := range n.GPUTotal {
|
||||
c := gpuCell{Index: i}
|
||||
if h, ok := held[gpuKey(n.ID, i)]; ok {
|
||||
c.Model, c.Replica = h.model, h.replica
|
||||
}
|
||||
var title []string
|
||||
title = append(title, fmt.Sprintf("%s GPU %d", n.ID, i))
|
||||
if c.Model != "" {
|
||||
title = append(title, c.Model+" ("+c.Replica+")")
|
||||
} else {
|
||||
title = append(title, "free")
|
||||
}
|
||||
s, sampled := sensorOf(n, i)
|
||||
if sampled {
|
||||
if s.MemoryTotalMB > 0 && s.UtilPercent >= 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f%% busy", s.UtilPercent))
|
||||
}
|
||||
if s.MemoryTotalMB > 0 {
|
||||
title = append(title, fmt.Sprintf("%.1f of %.0f GB", s.MemoryUsedMB/1024, s.MemoryTotalMB/1024))
|
||||
}
|
||||
if s.TempC > 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f °C", s.TempC))
|
||||
}
|
||||
if s.PowerW > 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f W", s.PowerW))
|
||||
}
|
||||
if len(s.Throttle) > 0 {
|
||||
title = append(title, "held back by "+strings.Join(s.Throttle, ", "))
|
||||
}
|
||||
title = append(title, s.Problems...)
|
||||
}
|
||||
c.Title = strings.Join(title, " · ")
|
||||
style, label := cellColour(m.By, s, sampled)
|
||||
if c.Model == "" {
|
||||
style += ";border-style:dashed"
|
||||
}
|
||||
// A card wearing out is outlined whatever the map is coloured
|
||||
// by: it is the one thing on the map that needs doing.
|
||||
if len(s.Problems) > 0 {
|
||||
style += ";border:2px solid #c0392b"
|
||||
c.Faulty = true
|
||||
}
|
||||
c.Style, c.Label = template.CSS(style), label
|
||||
row.Cells = append(row.Cells, c)
|
||||
}
|
||||
m.Rows = append(m.Rows, row)
|
||||
}
|
||||
if len(m.Rows) == 0 {
|
||||
return nil
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// cellColour is a cell's background, and the figure written in it.
|
||||
func cellColour(by string, s sensorView, sampled bool) (string, string) {
|
||||
const unknown = "background:var(--panel-2,#eee)"
|
||||
switch by {
|
||||
case "temp":
|
||||
if !sampled || s.TempC <= 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
// 30 °C blue to 90 °C red.
|
||||
f := clamp01((s.TempC - 30) / 60)
|
||||
return fmt.Sprintf("background:hsl(%.0f,70%%,55%%)", 220-220*f), fmt.Sprintf("%.0f", s.TempC)
|
||||
case "memory":
|
||||
if !sampled || s.MemoryTotalMB <= 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
f := clamp01(s.MemoryUsedMB / s.MemoryTotalMB)
|
||||
return fmt.Sprintf("background:hsl(265,45%%,%.0f%%)", 90-55*f), fmt.Sprintf("%.0f", 100*f)
|
||||
}
|
||||
if !sampled || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
f := clamp01(s.UtilPercent / 100)
|
||||
return fmt.Sprintf("background:hsl(140,%.0f%%,%.0f%%)", 15+50*f, 88-50*f), fmt.Sprintf("%.0f", s.UtilPercent)
|
||||
}
|
||||
|
||||
// idleHoldings are the replicas whose GPUs all read under 5% busy now:
|
||||
// capacity held and not asked for, the first place to find room.
|
||||
func idleHoldings(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, prefix string) []idleHolding {
|
||||
byID := map[string]adminapi.NodeView{}
|
||||
for _, n := range nodes {
|
||||
byID[n.ID] = n
|
||||
}
|
||||
var out []idleHolding
|
||||
for _, r := range reps {
|
||||
if r.State != "ready" {
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
if len(sh.GPUs) == 0 {
|
||||
continue
|
||||
}
|
||||
h := idleHolding{Model: r.Model, Replica: r.ID, Node: strings.TrimPrefix(sh.Node, prefix), GPUs: sh.GPUs}
|
||||
idle, known := true, true
|
||||
for _, g := range sh.GPUs {
|
||||
s, ok := sensorOf(byID[sh.Node], g)
|
||||
if !ok || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
|
||||
known = false
|
||||
break
|
||||
}
|
||||
h.MaxUtil = max(h.MaxUtil, s.UtilPercent)
|
||||
if s.UtilPercent >= 5 {
|
||||
idle = false
|
||||
}
|
||||
}
|
||||
if !known || !idle {
|
||||
continue
|
||||
}
|
||||
if r.IdleSec >= 0 {
|
||||
h.Idle = (time.Duration(r.IdleSec) * time.Second).String()
|
||||
}
|
||||
out = append(out, h)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return len(out[i].GPUs) > len(out[j].GPUs) })
|
||||
return out
|
||||
}
|
||||
|
||||
type holder struct{ model, replica string }
|
||||
|
||||
func heldBy(reps []adminapi.ReplicaView) map[string]holder {
|
||||
out := map[string]holder{}
|
||||
for _, r := range reps {
|
||||
switch r.State {
|
||||
case "stopped", "failed":
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
for _, g := range sh.GPUs {
|
||||
out[gpuKey(sh.Node, g)] = holder{r.Model, r.ID}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func gpuKey(node string, gpu int) string { return fmt.Sprintf("%s/%d", node, gpu) }
|
||||
|
||||
// sensorView is a card's last sample.
|
||||
type sensorView = struct {
|
||||
TempC, PowerW, UtilPercent, MemoryUsedMB, MemoryTotalMB float64
|
||||
Throttle []string
|
||||
Problems []string
|
||||
}
|
||||
|
||||
func sensorOf(n adminapi.NodeView, gpu int) (sensorView, bool) {
|
||||
if n.Load == nil {
|
||||
return sensorView{}, false
|
||||
}
|
||||
for _, s := range n.Load.GPUs {
|
||||
if s.Index == gpu {
|
||||
return sensorView{TempC: s.TempC, PowerW: s.PowerW, UtilPercent: s.UtilPercent,
|
||||
MemoryUsedMB: s.MemoryUsedMB, MemoryTotalMB: s.MemoryTotalMB, Throttle: s.Throttle,
|
||||
Problems: s.Wear.Problems()}, true
|
||||
}
|
||||
}
|
||||
return sensorView{}, false
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
)
|
||||
|
||||
// mapFleet is a node of four cards: chat holds two and is idle, code holds
|
||||
// one and is busy, one is free.
|
||||
type mapFleet struct{ fakeSource }
|
||||
|
||||
func (mapFleet) Nodes() []adminapi.NodeView {
|
||||
busy := func(i int, util, temp float64) inventory.GPUSensor {
|
||||
return inventory.GPUSensor{Index: i, UtilPercent: util, TempC: temp, MemoryUsedMB: 40000, MemoryTotalMB: 81559, PowerW: 300}
|
||||
}
|
||||
return []adminapi.NodeView{{ID: "gpu07", State: "ready", GPUTotal: 4, GPUFree: 1, Classes: []string{"h100-80g"},
|
||||
Load: &adminapi.LoadView{GPUs: []inventory.GPUSensor{busy(0, 1, 35), busy(1, 0, 34), busy(2, 93, 81), worn(busy(3, 0, 30))}}}}
|
||||
}
|
||||
|
||||
func worn(s inventory.GPUSensor) inventory.GPUSensor {
|
||||
s.Wear = &inventory.GPUWear{RemapPending: true, XIDs: 1, LastXID: 79, LastXIDText: "GPU has fallen off the bus."}
|
||||
return s
|
||||
}
|
||||
|
||||
func (mapFleet) Replicas(string) []adminapi.ReplicaView {
|
||||
return []adminapi.ReplicaView{
|
||||
{ID: "chat-1", Model: "chat", State: "ready", IdleSec: 1800, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{0, 1}}}},
|
||||
{ID: "code-1", Model: "code", State: "ready", IdleSec: 2, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{2}}}},
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClusterPageMapsEveryCardAndNamesTheIdleOnes(t *testing.T) {
|
||||
_, body := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if n := strings.Count(body, `class="gpu-cell"`); n != 4 {
|
||||
t.Errorf("%d cards on the map, want 4", n)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"gpu07 GPU 2 · code (code-1) · 93% busy",
|
||||
"gpu07 GPU 3 · free",
|
||||
"memory rows remapped, waiting for a reset · XID 79 (GPU has fallen off the bus.)",
|
||||
"border:2px solid #c0392b",
|
||||
"border-style:dashed",
|
||||
"Cards held and doing nothing",
|
||||
"<code>chat-1</code></td><td><code>gpu07</code>", "30m0s ago",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "<code>code-1</code></td><td><code>gpu07</code>") {
|
||||
t.Error("a busy replica was listed as idle")
|
||||
}
|
||||
_, temp := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/?gpumap=temp", nil)
|
||||
if !strings.Contains(temp, "<strong>temperature</strong>") || !strings.Contains(temp, ">81</span>") {
|
||||
t.Error("the map coloured by temperature does not say so, or show it")
|
||||
}
|
||||
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(cpu, "gpu-map") {
|
||||
t.Error("a fleet without GPUs has a GPU map")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachModelSaysWhatItGetsFromItsCards(t *testing.T) {
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{}}
|
||||
for i, vals := range [][2]float64{{120, 30}, {2, 2}, {1200, 400}, {80, 40}} {
|
||||
h.byExpr[efficiencyQueries[i].expr] = []promq.Series{ts("chat", vals[0], vals[0]), ts("code", vals[1], vals[1])}
|
||||
}
|
||||
got := loadEfficiency(context.Background(), h)
|
||||
if len(got) != 2 || got[0].Model != "chat" {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
c := got[0]
|
||||
if c.PerGPU != 60 || c.PerWh != 120*3600/1200.0 || c.Util != 80 {
|
||||
t.Errorf("chat: %+v", c)
|
||||
}
|
||||
if got[1].PerGPU != 15 {
|
||||
t.Errorf("code: %+v", got[1])
|
||||
}
|
||||
}
|
||||
@@ -40,8 +40,10 @@ type nodeEngine struct {
|
||||
// nodeCharts are the node's recent history, from the metrics store.
|
||||
type nodeCharts struct {
|
||||
CPU, Memory, Load, EngineCPU, EngineMemory, Queue TimeSeries
|
||||
Window string
|
||||
Err string
|
||||
// The GPUs', one line per card; empty on a node without them.
|
||||
GPUUtil, GPUMemory, GPUTemp, GPUPower TimeSeries
|
||||
Window string
|
||||
Err string
|
||||
}
|
||||
|
||||
// nodePanels are the charts, each a query scoped to this node. %s is the
|
||||
@@ -63,6 +65,23 @@ var nodePanels = []struct {
|
||||
func(c *nodeCharts, t TimeSeries) { c.EngineMemory = t }},
|
||||
}
|
||||
|
||||
// gpuPanels are the GPUs' charts, one line per card, for a node that has
|
||||
// them. Read from the heartbeat's samples, not from what the node said when
|
||||
// it registered.
|
||||
var gpuPanels = []struct {
|
||||
expr, unit string
|
||||
into func(*nodeCharts, TimeSeries)
|
||||
}{
|
||||
{`llmgw_gpu_utilization_percent{node=%s} / 100`, "%",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUUtil = t }},
|
||||
{`llmgw_gpu_memory_used_bytes{node=%s}`, "bytes",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUMemory = t }},
|
||||
{`llmgw_gpu_temperature_celsius{node=%s}`, "",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUTemp = t }},
|
||||
{`llmgw_gpu_power_watts{node=%s}`, "",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUPower = t }},
|
||||
}
|
||||
|
||||
// NodePage shows one node. Its charts come from the metrics store when one
|
||||
// is configured; the current figures come from the node's last heartbeat
|
||||
// either way.
|
||||
@@ -115,14 +134,14 @@ func NodePage(src Source, history History, window time.Duration, onAdminListener
|
||||
sort.Slice(d.Engines, func(i, j int) bool { return d.Engines[i].CPUCores > d.Engines[j].CPUCores })
|
||||
|
||||
if history != nil {
|
||||
d.Charts = loadNodeCharts(r.Context(), history, id, replicas, window, log)
|
||||
d.Charts = loadNodeCharts(r.Context(), history, id, replicas, d.Node.GPUTotal, window, log)
|
||||
}
|
||||
noStore(w)
|
||||
render(w, "node", d)
|
||||
}
|
||||
}
|
||||
|
||||
func loadNodeCharts(ctx context.Context, h History, node string, replicas []string,
|
||||
func loadNodeCharts(ctx context.Context, h History, node string, replicas []string, gpus int,
|
||||
window time.Duration, log *slog.Logger,
|
||||
) *nodeCharts {
|
||||
step := window / 60
|
||||
@@ -139,6 +158,19 @@ func loadNodeCharts(ctx context.Context, h History, node string, replicas []stri
|
||||
}
|
||||
p.into(out, LineChart(series, p.unit, 4))
|
||||
}
|
||||
// Every card its own line, up to the eight a node commonly has.
|
||||
if gpus > 0 {
|
||||
for _, p := range gpuPanels {
|
||||
series, err := h.RangeQuery(ctx, fmt.Sprintf(p.expr, quoted), "gpu", window, step)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for i := range series {
|
||||
series[i].Name = "GPU " + series[i].Name
|
||||
}
|
||||
p.into(out, LineChart(series, p.unit, max(gpus, 8)))
|
||||
}
|
||||
}
|
||||
// The engines' queues, for the replicas that are here now.
|
||||
if len(replicas) > 0 {
|
||||
re := make([]string, len(replicas))
|
||||
|
||||
@@ -205,3 +205,68 @@ func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// gpuSource is a node with two GPUs, one busy and one idle.
|
||||
type gpuSource struct{ fakeSource }
|
||||
|
||||
func (gpuSource) Nodes() []adminapi.NodeView {
|
||||
n := fakeSource{}.Nodes()[0]
|
||||
n.GPUTotal = 2
|
||||
n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
|
||||
GPUs: []inventory.GPUSensor{
|
||||
{Index: 0, TempC: 71, PowerW: 612, PowerLimitW: 700, UtilPercent: 97, MemUtilPercent: 64,
|
||||
MemoryUsedMB: 72909, MemoryTotalMB: 81559, SMClockMHz: 1980, ECCUncorrected: 0},
|
||||
{Index: 1, TempC: 30, PowerW: 70, PowerLimitW: 700, UtilPercent: 0, MemUtilPercent: 0,
|
||||
MemoryUsedMB: 512, MemoryTotalMB: 81559, SMClockMHz: 345, ECCUncorrected: 3},
|
||||
}}
|
||||
return []adminapi.NodeView{n}
|
||||
}
|
||||
|
||||
func TestAGPUNodesPageShowsHowBusyEachCardIsAndOverTime(t *testing.T) {
|
||||
q := func(expr string) string { return strings.Replace(expr, "%s", `"llmgw-node01"`, 1) }
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{
|
||||
q(`llmgw_gpu_utilization_percent{node=%s} / 100`): {ts("0", 0.9, 0.97), ts("1", 0, 0)},
|
||||
q(`llmgw_gpu_memory_used_bytes{node=%s}`): {ts("0", 7e10, 7.6e10)},
|
||||
q(`llmgw_gpu_temperature_celsius{node=%s}`): {ts("0", 65, 71)},
|
||||
q(`llmgw_gpu_power_watts{node=%s}`): {ts("0", 500, 612)},
|
||||
}}
|
||||
_, body := get(t, nodeMux(gpuSource{}, h), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, want := range []string{
|
||||
"97%", "memory 64%", "71.2 <span class=\"note\">of 80 GB", "1980 MHz",
|
||||
"3 uncorrected ECC errors",
|
||||
"GPU busy, by card", "GPU memory used, by card", "GPU temperature, °C", "GPU power, W", "GPU 0",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// A node without GPUs is not asked about them.
|
||||
h2 := &fakeHistory{}
|
||||
get(t, nodeMux(cpuFleet{}, h2), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, e := range h2.asked {
|
||||
if strings.Contains(e, "llmgw_gpu_") {
|
||||
t.Errorf("a node with no GPU was asked for %s", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClusterPageDrawsItsGPUsByClass(t *testing.T) {
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{
|
||||
`avg by (class) (llmgw_gpu_utilization_percent) / 100`: {ts("h100-80g", 0.2, 0.6)},
|
||||
`sum by (class) (llmgw_gpu_memory_used_bytes) / sum by (class) (llmgw_gpu_memory_total_bytes)`: {ts("h100-80g", 0.8, 0.9)},
|
||||
}}
|
||||
_, body := get(t, Dashboard(fakeSource{}, nil, "v1", time.Second, true, NavExtras{History: h, Window: time.Hour}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if !strings.Contains(body, "GPUs busy, by class") || !strings.Contains(body, "GPU memory in use, by class") {
|
||||
t.Error("a fleet with GPUs has no GPU charts")
|
||||
}
|
||||
h2 := &fakeHistory{}
|
||||
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{History: h2, Window: time.Hour}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(cpu, "GPUs busy") {
|
||||
t.Error("a fleet without GPUs was drawn GPU charts")
|
||||
}
|
||||
for _, e := range h2.asked {
|
||||
if strings.Contains(e, "llmgw_gpu_") {
|
||||
t.Errorf("a fleet without GPUs was asked for %s", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -862,3 +862,10 @@ table.rows td.warn { color: var(--warn); cursor: help; }
|
||||
.held-card .adapters h4 { font-size: .9rem; margin: 0 0 .35rem; display: flex; gap: .75rem; align-items: baseline; }
|
||||
.held-card .adapter-form .field-row { align-items: flex-end; flex-wrap: wrap; gap: .5rem 1rem; }
|
||||
.held-card .adapter-form select { display: block; margin-top: .25rem; max-width: 28rem; }
|
||||
|
||||
/* The GPU map: a square per card, a row per node. */
|
||||
.gpu-map { display: flex; flex-direction: column; gap: 3px; margin: 6px 0; overflow-x: auto; }
|
||||
.gpu-row { display: flex; align-items: center; gap: 3px; }
|
||||
.gpu-node { width: 7em; flex: none; font-size: 12px; color: var(--muted, #666); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.gpu-cell { width: 26px; height: 26px; flex: none; border: 1px solid rgba(0,0,0,.18); border-radius: 4px;
|
||||
font-size: 10px; line-height: 26px; text-align: center; color: rgba(0,0,0,.7); cursor: default; }
|
||||
@@ -63,6 +63,21 @@
|
||||
<p class="empty">Nothing recorded in this window.</p>
|
||||
{{end}}
|
||||
|
||||
{{with .GPUHours}}
|
||||
<h2>GPU time, by owner</h2>
|
||||
{{/* What each owner's models held, whether or not anybody asked them
|
||||
anything: a model kept loaded holds its cards all the same, and that is
|
||||
the time a GPU quota is about. */}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" $.GPUHeld "title" "GPUs held, by owner" "note" "The cards held by the replicas of each owner's models: models whose owner is a part of the quota tree."}}
|
||||
</div>
|
||||
<table class="rows">
|
||||
<tr><th>Owner</th><th class="num">GPU-hours</th><th class="num">Held now</th></tr>
|
||||
{{range .}}<tr><td>{{.Owner}}</td><td class="num">{{printf "%.1f" .Hours}}</td><td class="num">{{printf "%.0f" .Now}}</td></tr>{{end}}
|
||||
</table>
|
||||
<p class="note">From the metrics store, which keeps its history for as long as it is set to (fifteen days by default); a model with no owner is counted under “no owner”.</p>
|
||||
{{end}}
|
||||
|
||||
{{/* The charts fold their tail; the table does not. Both are here because they
|
||||
answer different questions, and the one that is easy to get wrong is
|
||||
thinking the four coloured lines are all of it. */}}
|
||||
|
||||
@@ -49,6 +49,36 @@
|
||||
</div>
|
||||
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
|
||||
|
||||
{{with .GPUMap}}
|
||||
<h2>GPUs <span class="opt">by
|
||||
{{range .Modes}}{{if eq .Key $.GPUMap.By}}<strong>{{.Label}}</strong>{{else}}<a href="?gpumap={{.Key}}">{{.Label}}</a>{{end}} {{end}}</span></h2>
|
||||
<div class="gpu-map">
|
||||
{{range .Rows}}<div class="gpu-row"><span class="gpu-node">{{.Node}}</span>{{range .Cells}}<span class="gpu-cell" style="{{.Style}}" title="{{.Title}}">{{.Label}}</span>{{end}}</div>
|
||||
{{end}}</div>
|
||||
<p class="note">{{.Legend}} A red outline is a card wearing out — XID errors, memory rows remapped or waiting for a reset, NVLink errors. The figure is the one coloured by; the rest of a card is under the pointer.</p>
|
||||
{{end}}
|
||||
{{with .Efficiency}}
|
||||
<h3>What each model gets from its cards <span class="opt">last 15 minutes</span></h3>
|
||||
{{/* Per card, because a model on eight cards making twice the tokens of one
|
||||
on two is the worse of the two; per watt-hour, because that is what the
|
||||
room is paying. */}}
|
||||
<table class="rows"><tr><th>Model</th><th class="num">GPUs</th><th class="num">Output tokens/s</th><th class="num">Per GPU</th><th class="num">Busy</th><th class="num">Draw</th><th class="num">Tokens per Wh</th></tr>
|
||||
{{range .}}<tr><td>{{.Model}}</td><td class="num">{{printf "%.0f" .GPUs}}</td><td class="num">{{printf "%.1f" .TokensPerSec}}</td>
|
||||
<td class="num">{{printf "%.1f" .PerGPU}}</td><td class="num">{{printf "%.0f" .Util}}%</td>
|
||||
<td class="num">{{if .Watts}}{{printf "%.0f" .Watts}} W{{else}}—{{end}}</td><td class="num">{{if .PerWh}}{{printf "%.0f" .PerWh}}{{else}}—{{end}}</td></tr>
|
||||
{{end}}</table>
|
||||
{{end}}
|
||||
{{with .IdleGPUs}}
|
||||
<h3>Cards held and doing nothing</h3>
|
||||
{{/* The first place to find room: replicas whose every card reads under 5%
|
||||
busy now. A model kept warm on purpose is here too; whether it should
|
||||
be is the operator's call. */}}
|
||||
<table class="rows"><tr><th>Model</th><th>Replica</th><th>Node</th><th class="num">GPUs</th><th class="num">Busiest card</th><th>Last request</th></tr>
|
||||
{{range .}}<tr><td>{{.Model}}</td><td class="clip"><code>{{.Replica}}</code></td><td><code>{{.Node}}</code></td>
|
||||
<td class="num">{{len .GPUs}}</td><td class="num">{{printf "%.0f" .MaxUtil}}%</td><td>{{if .Idle}}{{.Idle}} ago{{else}}<span class="note">not known</span>{{end}}</td></tr>
|
||||
{{end}}</table>
|
||||
{{end}}
|
||||
|
||||
<h2>Nodes</h2>
|
||||
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
|
||||
{{if .Nodes}}
|
||||
@@ -72,7 +102,7 @@
|
||||
together. Only a GPU node has one; changing it is for an
|
||||
administrator, on the page reached through a session. */}}
|
||||
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{if and $.Menu.CSRF .EngineBuilds}}
|
||||
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
|
||||
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
|
||||
@@ -213,6 +243,14 @@ unused nodes, which is what decides whether such a replica can land.</p>
|
||||
{{template "tschart" dict "t" .Ready "title" "Replicas ready, by model"
|
||||
"note" "A model deliberately stopped has no line at all, which is why an on-demand catalogue looks quiet here."}}
|
||||
</div>
|
||||
{{if $.AnyGPUs}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .GPUBusy "title" "GPUs busy, by class"
|
||||
"note" "The average card of each class. Low here with replicas placed is capacity held and not asked for; each node's page has its cards one by one."}}
|
||||
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory in use, by class"
|
||||
"note" "Engines reserve most of a card's memory when they start, so this follows replicas placed more than requests served."}}
|
||||
</div>
|
||||
{{end}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica"
|
||||
"note" "Requests accepted and not yet started. The first thing to move when a replica is falling behind."}}
|
||||
|
||||
@@ -35,13 +35,18 @@
|
||||
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
|
||||
{{with .Load}}{{if .GPUs}}
|
||||
<h2>GPUs</h2>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by, and wear</th></tr>
|
||||
{{range .GPUs}}<tr>
|
||||
<td class="num">{{.Index}}</td>
|
||||
{{/* From the last sample; a dash where the driver or an older agent did
|
||||
not say, rather than a zero nobody measured. */}}
|
||||
<td class="load">{{if and .MemoryTotalMB (ge .UtilPercent 0.0)}}{{template "minibar" pctbar .UtilPercent}}<span class="load-v">{{printf "%.0f" .UtilPercent}}%{{if ge .MemUtilPercent 0.0}} <span class="note">memory {{printf "%.0f" .MemUtilPercent}}%</span>{{end}}</span>{{else}}—{{end}}</td>
|
||||
<td class="load">{{if .MemoryTotalMB}}{{template "minibar" pctbar (percent .MemoryUsedMB .MemoryTotalMB)}}<span class="load-v">{{printf "%.1f" (gib .MemoryUsedMB)}} <span class="note">of {{printf "%.0f" (gib .MemoryTotalMB)}} GB</span></span>{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
|
||||
<td class="num">{{if .SMClockMHz}}{{printf "%.0f" .SMClockMHz}} MHz{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{.PCIeReplays}} PCIe retries</span>{{end}}{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{end}}{{end}}
|
||||
{{end}}
|
||||
@@ -120,6 +125,16 @@ traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS
|
||||
{{template "tschart" dict "t" .EngineCPU "title" "Cores busy, by model" "note" "Each engine's process group, the way top counts it."}}
|
||||
{{template "tschart" dict "t" .EngineMemory "title" "Engine memory, by model" "note" "Resident memory, including the weights mapped from disk. The kernel can take those pages back, so this can exceed RAM in use."}}
|
||||
</div>
|
||||
{{if $.Node.GPUTotal}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .GPUUtil "title" "GPU busy, by card" "note" "Of each sample period, the share a kernel was running. A card held by a replica and idle here is one nobody is asking."}}
|
||||
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory used, by card" "note" "Framebuffer in use: the weights and the KV cache an engine reserved, and anything not llmgw's."}}
|
||||
</div>
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .GPUTemp "title" "GPU temperature, °C" "note" "Throttling usually starts in the high 80s; see what holds a card back in the table above."}}
|
||||
{{template "tschart" dict "t" .GPUPower "title" "GPU power, W" "note" "Draw against the card's enforced limit; at the limit is sw_power_cap."}}
|
||||
</div>
|
||||
{{end}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .Load "title" "Load average" "note" "Above the core count means work is waiting for a CPU."}}
|
||||
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica" "note" "Requests accepted and not yet started, on the replicas here now."}}
|
||||
|
||||
@@ -110,6 +110,8 @@ var funcs = template.FuncMap{
|
||||
return p
|
||||
},
|
||||
"ago": ago,
|
||||
// gib is mebibytes as gibibytes, for a GPU's memory.
|
||||
"gib": func(mib float64) float64 { return mib / 1024 },
|
||||
// percent is part of whole, 0 to 100, for a bar.
|
||||
"percent": func(part, whole float64) float64 {
|
||||
if whole <= 0 {
|
||||
|
||||
Reference in new issue
Block a user