Compare commits
22
Commits
v0.10.6
...
v0.10.7-rc.11
No files matched your search
+302
@@ -20,6 +20,308 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
From the field report in issue #2, after a deployment on GPU nodes.
|
||||
|
||||
### GPU metrics that move, and charts of them
|
||||
|
||||
A GPU's utilisation and memory in use were read once, when the agent
|
||||
started, and published from then on: a card busy since looked as idle as it
|
||||
was at registration, and one idle since as busy. They are now sampled with
|
||||
the temperatures and power every heartbeat (ten seconds at most), with the
|
||||
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
|
||||
An agent too old to send them still has its registration figures published,
|
||||
and a figure the driver does not give is left out rather than shown as zero.
|
||||
|
||||
- **New series**: `llmgw_gpu_memory_utilization_percent`,
|
||||
`llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
|
||||
`llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
|
||||
sample.
|
||||
- **The node page** shows each card's utilisation, memory, clock and ECC
|
||||
errors in its table, and charts of utilisation, memory, temperature and
|
||||
power, one line per card.
|
||||
- **The cluster page** charts the fleet's GPUs by class: average utilisation
|
||||
and memory in use.
|
||||
- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
|
||||
utilisation, memory, memory bandwidth, temperature, power against the
|
||||
limit, clock, and what holds cards back.
|
||||
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
|
||||
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
|
||||
- **The cluster page's GPU map**: every card a square, a row per node,
|
||||
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
|
||||
with what holds it and its figures under the pointer; free cards dashed.
|
||||
- **Cards held and doing nothing**: the replicas whose every card reads under
|
||||
5% busy, with when they were last asked — the first place to find room.
|
||||
- **What each model gets from its cards**, on the cluster page and in
|
||||
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
|
||||
draw and average utilisation, over the last quarter hour.
|
||||
- **What wears a card out**, read every minute: XID errors from the kernel's
|
||||
log (counted from when the agent started), memory rows remapped or pages
|
||||
retired and whether either waits for a reset, a card out of spare rows,
|
||||
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
|
||||
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
|
||||
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
|
||||
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
|
||||
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
|
||||
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
|
||||
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
|
||||
`GPUNVLinkErrorsClimbing`.
|
||||
- **GPU time by owner** on the usage page: the cards each owner's models
|
||||
held over the window, charted and summed into GPU-hours, whether or not the
|
||||
models were asked anything — the time a GPU quota is about. From the
|
||||
metrics store, as far back as it keeps history.
|
||||
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
|
||||
each card and whose quota it counts against, for joining the GPU series to
|
||||
models and tenants.
|
||||
|
||||
### People, limits and usage are one page
|
||||
|
||||
Managing a team meant its members on the people page, its quota on the quotas
|
||||
page and its spend on the usage page, each linking to the others. They are now
|
||||
one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
|
||||
each at the address it had before, so links and bookmarks still land. The menu
|
||||
has one entry for them, People.
|
||||
|
||||
Choosing an organization, team, project or person in the tree, or a person in
|
||||
the account list, opens a panel beside it with everything about that one
|
||||
thing: its own limits and what it is held to from above, spend against its
|
||||
budget, its per-member limit, its capacity quota, and its requests, tokens
|
||||
and spend over the last week, drawn. Its limits can be set or changed there,
|
||||
and the answer comes back to the same panel.
|
||||
|
||||
The administrator's menu no longer has an Organizations entry beside it: `/org`
|
||||
and `/org/<id>` send an administrator to that organization's panel here. The
|
||||
page stays as it was for an organization's managers, who cannot see this one.
|
||||
|
||||
### The chat page says whether a GPU or a CPU answered
|
||||
|
||||
A model can have replicas on a card and on CPUs at once, and the two answer
|
||||
at very different speeds; the chat page gave no way to tell which one a reply
|
||||
came from. The picker now says where each model runs — `GPU`, `CPU` or
|
||||
`GPU + CPU` — and the line under each reply names what answered it: `GPU
|
||||
6000-24g` or `CPU`, beside its tokens a second.
|
||||
|
||||
Every answer through the gateway carries the same as `x-llmgw-hardware`:
|
||||
`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
|
||||
none.
|
||||
|
||||
### A card shared with something else is judged every heartbeat
|
||||
|
||||
A card with another process on it when its agent started was kept out of
|
||||
placement, and stayed out — "5014 MiB already allocated by another process" —
|
||||
after that process let go of it, until the agent was restarted. Found on a
|
||||
node shared with ComfyUI. The other way round, a tenant arriving after
|
||||
registration was never seen, and a card the agent's own engine held when the
|
||||
agent restarted read as someone else's.
|
||||
|
||||
The agent now reads every card's memory per process on each heartbeat,
|
||||
leaves out what its own engines hold, and sends the rest; the control plane
|
||||
keeps a card out of placement on that reading, and lets it back in when it
|
||||
falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
|
||||
agent too old to send it keeps the reading from registration.
|
||||
|
||||
### A GPU placement no longer runs on the CPU unnoticed
|
||||
|
||||
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
|
||||
where the loader looks), came up on the CPU, answered at CPU speed, and was
|
||||
reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||||
|
||||
- **Installing** the CUDA build asks it which devices it sees
|
||||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||||
own `ldd` could not say it: the backend is opened at run time.
|
||||
- **After the upgrade**, engines the agent finds already running are held to
|
||||
the same check when it adopts them: one serving a GPU placement on the CPU
|
||||
is stopped, and its replacement starts through the check. The installed
|
||||
CUDA build is checked again within ten minutes, and installed afresh if it
|
||||
sees no GPU.
|
||||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||||
included). One that does not is stopped and reported failed: "placed on GPU
|
||||
0 but computing on the CPU", with the devices the engine lists. This covers
|
||||
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
|
||||
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
|
||||
when the engine sees no GPU, and otherwise prints the devices it computes on.
|
||||
|
||||
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
|
||||
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
|
||||
node with two kinds of GPU, a shard could get a different card from the one it
|
||||
was given.
|
||||
|
||||
### A model of many files, mirrored whole for vLLM
|
||||
|
||||
vLLM loads a directory — shards, config.json, the tokenizer — and does not
|
||||
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
|
||||
shards alone, a node could be handed a version still downloading or missing a
|
||||
shard, and a repository's page given as a model's source was fetched as one
|
||||
file of HTML.
|
||||
|
||||
- **The explorer fetches a safetensors set with what the engine reads beside
|
||||
it**: config.json, generation_config.json, the tokenizer files, the shard
|
||||
index, the quantisation config. Never code: a `.py` would be the engine
|
||||
running what the repository ships, which is to be chosen on purpose.
|
||||
- **A version is offered to nodes only whole.** The mirror refuses its
|
||||
manifest (409, with what is missing) while a file is still downloading, or
|
||||
while a shard its `model.safetensors.index.json` names, or a part of a
|
||||
split (`-00002-of-00004`), is absent. A node reports "not complete on the
|
||||
mirror yet" and is tried again.
|
||||
- **Checksums go with the files.** The mirror keeps each file's sha256 beside
|
||||
it and the manifest carries it; a node checks every file of a set against
|
||||
it before using the set, and fetches one that differs again.
|
||||
- **`llmgw mirror add` takes several URLs** into one version.
|
||||
- **A Hugging Face repository page given as a source** is read as the
|
||||
repository id for a model vLLM or SGLang runs, which the engine downloads
|
||||
itself (the node then needs to reach Hugging Face), and refused for one
|
||||
llama.cpp runs, which needs a file.
|
||||
|
||||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||||
such a model, as before.
|
||||
|
||||
### Signing up with an address that has an account
|
||||
|
||||
On PostgreSQL, signing up with an address already registered showed
|
||||
"something went wrong; try again": the store recognised a duplicate by the
|
||||
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
|
||||
code 23505. It now knows both, and the address is answered as on SQLite —
|
||||
with the same page as a new sign-up, so the page cannot be used to find out
|
||||
who has an account. That page now also says what to do if the address had
|
||||
one: sign in, or ask an operator to reset the password.
|
||||
|
||||
An error the page does not explain says the account was not created and
|
||||
gives a reference (`err-…`), logged beside the error, for an operator to find
|
||||
it; the database's own words stay in the log.
|
||||
|
||||
### An adapter no fabric uses does not keep a node out
|
||||
|
||||
`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
|
||||
`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
|
||||
added for single-node inference, with no fabric given, that refused it for
|
||||
RDMA nobody had asked for.
|
||||
|
||||
- Those are failures only for ports a declared fabric uses, and only when a
|
||||
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
|
||||
- Which fabric an undeclared InfiniBand port is on, and whether it has a
|
||||
subnet manager LID, are notes rather than warnings; `node add` still warns
|
||||
once that adapters are up with no fabric given.
|
||||
- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
|
||||
name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
|
||||
written afterwards.
|
||||
|
||||
### A restart with no room beside it
|
||||
|
||||
`ctl restart` places each replacement beside the replica it replaces. With
|
||||
no room for it — one GPU, every card that fits held by the replica itself —
|
||||
the old replica was stopped only after the drain grace period, five minutes
|
||||
of "restart replacing" with nothing saying why, which read as a restart that
|
||||
never happened; scaling to zero and back was the way out. And a replacement
|
||||
that fit nowhere even with the old replica gone was stopped for all the same,
|
||||
leaving the model with nothing. Now:
|
||||
|
||||
- **The replica is the only thing in the way**, found by placing the
|
||||
replacement on a copy of the inventory without it: stopped at once, the
|
||||
replacement taking its place.
|
||||
- **The replacement fits nowhere even then**: the restart is given up and the
|
||||
replica keeps serving. `ctl replicas` shows `restart refused:` and why.
|
||||
- **Room being made** — a lower class preempted, an idle on-demand replica
|
||||
stopping — is waited for, as before.
|
||||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||||
|
||||
### Every GPU node gets engine builds it can run, by itself
|
||||
|
||||
Which build of an engine a node installs follows its hardware — the CPU
|
||||
architecture, the driver, and the oldest GPU on it — from what each project
|
||||
publishes and how it was compiled. Each engine has a list of builds, best
|
||||
first; the node takes the first its hardware meets, and the first of those
|
||||
that installs and is seen to compute on the GPU. A build the table is wrong
|
||||
about costs its download, and the next one is tried.
|
||||
|
||||
| | llama.cpp | vLLM |
|
||||
|---|---|---|
|
||||
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
|
||||
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
|
||||
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
|
||||
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
|
||||
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
|
||||
| No NVIDIA GPU | the CPU build | none |
|
||||
|
||||
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
|
||||
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
|
||||
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
|
||||
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
|
||||
starts at 7.5.
|
||||
|
||||
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
|
||||
column shows the version installed, and the installation's detail says why
|
||||
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
|
||||
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
|
||||
vLLM is not placed on a V100 by accident.
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation for vLLM, the devices llama.cpp lists for its GPU builds.
|
||||
- **The build can still be chosen per node** on the cluster page (auto, CUDA
|
||||
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
|
||||
at once.
|
||||
- **Whether an engine computes on its GPUs is read from every process
|
||||
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
|
||||
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
|
||||
`CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
### Engines per node, and nodes that do not run them all
|
||||
|
||||
A cluster with older GPUs could not add them cleanly: `node add` refused a
|
||||
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
|
||||
and `--ignore-checks` passed over every other problem with it.
|
||||
|
||||
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
|
||||
node joins without it and serves what it does run. `node add` checks the
|
||||
vLLM the control plane installs, not the binary's own default.
|
||||
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none — for exceptions; the
|
||||
build each GPU generation needs is chosen without it (above).
|
||||
- **The requirements follow the version**, as the table above sets out; the
|
||||
threshold had been CUDA 13's for every version.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
|
||||
### Serving from the mirror says where a model will run
|
||||
|
||||
The serve form put every GGUF model on the CPU: it offered no choice, and its
|
||||
"Profile name" field, pre-filled "cpu", read as though renaming it would move
|
||||
the model to a GPU. Placement is decided by `gpu_class`, which the form never
|
||||
set. Now:
|
||||
|
||||
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
|
||||
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
|
||||
cards is chosen.
|
||||
- **GPUs per replica** sets how many cards a replica takes on one node
|
||||
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
|
||||
how many a llama.cpp replica serves at once.
|
||||
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
|
||||
typed.
|
||||
- A line under the form says what the model will run as — engine, CPU or how
|
||||
many of which GPU, memory, context pool and how it is divided — and turns
|
||||
red when the CPU is chosen with GPUs free. The message after adding repeats
|
||||
it.
|
||||
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
|
||||
anyway** is ticked.
|
||||
|
||||
### llama.cpp on several GPUs of one node
|
||||
|
||||
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
|
||||
placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||||
placed — the replica stayed unplaced or ran on one card. It is now read as
|
||||
`gpus_per_node`, so a profile written that way is placed on that many cards
|
||||
after the upgrade, rather than dropped; only both given and disagreeing, or
|
||||
`nodes` above one, is refused.
|
||||
|
||||
## v0.10.6 — 2026-10-01
|
||||
|
||||
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
|
||||
|
||||
+248
-13
@@ -1550,7 +1550,36 @@ type GPUSensor struct {
|
||||
PowerLimitW float64 `protobuf:"fixed64,5,opt,name=power_limit_w,json=powerLimitW,proto3" json:"power_limit_w,omitempty"`
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
|
||||
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
|
||||
// How busy the card is, sampled with the rest, rather than read once at
|
||||
// registration. -1 is not reported by the driver. memory_total_mb is set
|
||||
// by every agent that sends these, so zero there is an older agent.
|
||||
UtilizationPercent float64 `protobuf:"fixed64,7,opt,name=utilization_percent,json=utilizationPercent,proto3" json:"utilization_percent,omitempty"`
|
||||
MemoryUtilizationPercent float64 `protobuf:"fixed64,8,opt,name=memory_utilization_percent,json=memoryUtilizationPercent,proto3" json:"memory_utilization_percent,omitempty"`
|
||||
MemoryUsedMb float64 `protobuf:"fixed64,9,opt,name=memory_used_mb,json=memoryUsedMb,proto3" json:"memory_used_mb,omitempty"`
|
||||
MemoryTotalMb float64 `protobuf:"fixed64,10,opt,name=memory_total_mb,json=memoryTotalMb,proto3" json:"memory_total_mb,omitempty"`
|
||||
SmClockMhz float64 `protobuf:"fixed64,11,opt,name=sm_clock_mhz,json=smClockMhz,proto3" json:"sm_clock_mhz,omitempty"`
|
||||
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
|
||||
EccUncorrected float64 `protobuf:"fixed64,12,opt,name=ecc_uncorrected,json=eccUncorrected,proto3" json:"ecc_uncorrected,omitempty"`
|
||||
// What wears the card out, read every minute; has_wear says these are
|
||||
// present. Counts of -1 are not reported for this card.
|
||||
HasWear bool `protobuf:"varint,13,opt,name=has_wear,json=hasWear,proto3" json:"has_wear,omitempty"`
|
||||
PcieReplays int64 `protobuf:"varint,14,opt,name=pcie_replays,json=pcieReplays,proto3" json:"pcie_replays,omitempty"`
|
||||
RetiredPagesSbe int64 `protobuf:"varint,15,opt,name=retired_pages_sbe,json=retiredPagesSbe,proto3" json:"retired_pages_sbe,omitempty"`
|
||||
RetiredPagesDbe int64 `protobuf:"varint,16,opt,name=retired_pages_dbe,json=retiredPagesDbe,proto3" json:"retired_pages_dbe,omitempty"`
|
||||
RetirementPending bool `protobuf:"varint,17,opt,name=retirement_pending,json=retirementPending,proto3" json:"retirement_pending,omitempty"`
|
||||
RemappedRowsCorrectable int64 `protobuf:"varint,18,opt,name=remapped_rows_correctable,json=remappedRowsCorrectable,proto3" json:"remapped_rows_correctable,omitempty"`
|
||||
RemappedRowsUncorrectable int64 `protobuf:"varint,19,opt,name=remapped_rows_uncorrectable,json=remappedRowsUncorrectable,proto3" json:"remapped_rows_uncorrectable,omitempty"`
|
||||
RemapPending bool `protobuf:"varint,20,opt,name=remap_pending,json=remapPending,proto3" json:"remap_pending,omitempty"`
|
||||
RemapFailure bool `protobuf:"varint,21,opt,name=remap_failure,json=remapFailure,proto3" json:"remap_failure,omitempty"`
|
||||
NvlinkCrcErrors int64 `protobuf:"varint,22,opt,name=nvlink_crc_errors,json=nvlinkCrcErrors,proto3" json:"nvlink_crc_errors,omitempty"`
|
||||
NvlinkReplayErrors int64 `protobuf:"varint,23,opt,name=nvlink_replay_errors,json=nvlinkReplayErrors,proto3" json:"nvlink_replay_errors,omitempty"`
|
||||
NvlinkRecoveryErrors int64 `protobuf:"varint,24,opt,name=nvlink_recovery_errors,json=nvlinkRecoveryErrors,proto3" json:"nvlink_recovery_errors,omitempty"`
|
||||
// xid_errors are the driver's error events since the agent started.
|
||||
XidErrors int64 `protobuf:"varint,25,opt,name=xid_errors,json=xidErrors,proto3" json:"xid_errors,omitempty"`
|
||||
LastXid int32 `protobuf:"varint,26,opt,name=last_xid,json=lastXid,proto3" json:"last_xid,omitempty"`
|
||||
LastXidText string `protobuf:"bytes,27,opt,name=last_xid_text,json=lastXidText,proto3" json:"last_xid_text,omitempty"`
|
||||
LastXidUnix int64 `protobuf:"varint,28,opt,name=last_xid_unix,json=lastXidUnix,proto3" json:"last_xid_unix,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -1627,6 +1656,160 @@ func (x *GPUSensor) GetThrottle() []string {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetUtilizationPercent() float64 {
|
||||
if x != nil {
|
||||
return x.UtilizationPercent
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryUtilizationPercent() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryUtilizationPercent
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryUsedMb() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryUsedMb
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryTotalMb() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryTotalMb
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetSmClockMhz() float64 {
|
||||
if x != nil {
|
||||
return x.SmClockMhz
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetEccUncorrected() float64 {
|
||||
if x != nil {
|
||||
return x.EccUncorrected
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetHasWear() bool {
|
||||
if x != nil {
|
||||
return x.HasWear
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetPcieReplays() int64 {
|
||||
if x != nil {
|
||||
return x.PcieReplays
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetiredPagesSbe() int64 {
|
||||
if x != nil {
|
||||
return x.RetiredPagesSbe
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetiredPagesDbe() int64 {
|
||||
if x != nil {
|
||||
return x.RetiredPagesDbe
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetirementPending() bool {
|
||||
if x != nil {
|
||||
return x.RetirementPending
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemappedRowsCorrectable() int64 {
|
||||
if x != nil {
|
||||
return x.RemappedRowsCorrectable
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemappedRowsUncorrectable() int64 {
|
||||
if x != nil {
|
||||
return x.RemappedRowsUncorrectable
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemapPending() bool {
|
||||
if x != nil {
|
||||
return x.RemapPending
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemapFailure() bool {
|
||||
if x != nil {
|
||||
return x.RemapFailure
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkCrcErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkCrcErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkReplayErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkReplayErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkRecoveryErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkRecoveryErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetXidErrors() int64 {
|
||||
if x != nil {
|
||||
return x.XidErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXid() int32 {
|
||||
if x != nil {
|
||||
return x.LastXid
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXidText() string {
|
||||
if x != nil {
|
||||
return x.LastXidText
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXidUnix() int64 {
|
||||
if x != nil {
|
||||
return x.LastXidUnix
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
type ShardLoad struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -2153,9 +2336,17 @@ type Heartbeat struct {
|
||||
GpuFabric []*GPUFabricState `protobuf:"bytes,16,rep,name=gpu_fabric,json=gpuFabric,proto3" json:"gpu_fabric,omitempty"`
|
||||
Imex *IMEX `protobuf:"bytes,17,opt,name=imex,proto3" json:"imex,omitempty"`
|
||||
// cpu_count as in Register, for a VM given more CPUs while it runs.
|
||||
CpuCount int32 `protobuf:"varint,18,opt,name=cpu_count,json=cpuCount,proto3" json:"cpu_count,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
CpuCount int32 `protobuf:"varint,18,opt,name=cpu_count,json=cpuCount,proto3" json:"cpu_count,omitempty"`
|
||||
// gpu_foreign_memory_mb, indexed by GPU, is what processes that are not
|
||||
// this agent's shards hold on each card, read afresh every heartbeat. What
|
||||
// keeps a card out of placement for another tenant is decided from this
|
||||
// rather than from registration: a card that had something else on it when
|
||||
// the agent started is not taken forever, and one that gains a tenant later
|
||||
// is not handed out. Empty from an agent that does not measure it, or from
|
||||
// one that could not this time.
|
||||
GpuForeignMemoryMb []int32 `protobuf:"varint,19,rep,packed,name=gpu_foreign_memory_mb,json=gpuForeignMemoryMb,proto3" json:"gpu_foreign_memory_mb,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Heartbeat) Reset() {
|
||||
@@ -2314,6 +2505,13 @@ func (x *Heartbeat) GetCpuCount() int32 {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *Heartbeat) GetGpuForeignMemoryMb() []int32 {
|
||||
if x != nil {
|
||||
return x.GpuForeignMemoryMb
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
// versions. Nothing more.
|
||||
//
|
||||
@@ -2370,9 +2568,12 @@ func (x *Provision) GetEngines() []*EngineWant {
|
||||
|
||||
// EngineWant names one engine at one version.
|
||||
type EngineWant struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
|
||||
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
|
||||
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
|
||||
// variant is the build to install, when one is chosen for the node rather
|
||||
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
|
||||
Variant string `protobuf:"bytes,3,opt,name=variant,proto3" json:"variant,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -2421,6 +2622,13 @@ func (x *EngineWant) GetVersion() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *EngineWant) GetVariant() string {
|
||||
if x != nil {
|
||||
return x.Variant
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ProvisionStatus is where one engine's installation stands on a node.
|
||||
type ProvisionStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -3989,14 +4197,39 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
" \x01(\x01R\tcpuPowerW\x12 \n" +
|
||||
"\finlet_temp_c\x18\v \x01(\x01R\n" +
|
||||
"inletTempC\x12'\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xb5\x01\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xde\b\n" +
|
||||
"\tGPUSensor\x12\x14\n" +
|
||||
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x15\n" +
|
||||
"\x06temp_c\x18\x02 \x01(\x01R\x05tempC\x12\"\n" +
|
||||
"\rmemory_temp_c\x18\x03 \x01(\x01R\vmemoryTempC\x12\x17\n" +
|
||||
"\apower_w\x18\x04 \x01(\x01R\x06powerW\x12\"\n" +
|
||||
"\rpower_limit_w\x18\x05 \x01(\x01R\vpowerLimitW\x12\x1a\n" +
|
||||
"\bthrottle\x18\x06 \x03(\tR\bthrottle\"m\n" +
|
||||
"\bthrottle\x18\x06 \x03(\tR\bthrottle\x12/\n" +
|
||||
"\x13utilization_percent\x18\a \x01(\x01R\x12utilizationPercent\x12<\n" +
|
||||
"\x1amemory_utilization_percent\x18\b \x01(\x01R\x18memoryUtilizationPercent\x12$\n" +
|
||||
"\x0ememory_used_mb\x18\t \x01(\x01R\fmemoryUsedMb\x12&\n" +
|
||||
"\x0fmemory_total_mb\x18\n" +
|
||||
" \x01(\x01R\rmemoryTotalMb\x12 \n" +
|
||||
"\fsm_clock_mhz\x18\v \x01(\x01R\n" +
|
||||
"smClockMhz\x12'\n" +
|
||||
"\x0fecc_uncorrected\x18\f \x01(\x01R\x0eeccUncorrected\x12\x19\n" +
|
||||
"\bhas_wear\x18\r \x01(\bR\ahasWear\x12!\n" +
|
||||
"\fpcie_replays\x18\x0e \x01(\x03R\vpcieReplays\x12*\n" +
|
||||
"\x11retired_pages_sbe\x18\x0f \x01(\x03R\x0fretiredPagesSbe\x12*\n" +
|
||||
"\x11retired_pages_dbe\x18\x10 \x01(\x03R\x0fretiredPagesDbe\x12-\n" +
|
||||
"\x12retirement_pending\x18\x11 \x01(\bR\x11retirementPending\x12:\n" +
|
||||
"\x19remapped_rows_correctable\x18\x12 \x01(\x03R\x17remappedRowsCorrectable\x12>\n" +
|
||||
"\x1bremapped_rows_uncorrectable\x18\x13 \x01(\x03R\x19remappedRowsUncorrectable\x12#\n" +
|
||||
"\rremap_pending\x18\x14 \x01(\bR\fremapPending\x12#\n" +
|
||||
"\rremap_failure\x18\x15 \x01(\bR\fremapFailure\x12*\n" +
|
||||
"\x11nvlink_crc_errors\x18\x16 \x01(\x03R\x0fnvlinkCrcErrors\x120\n" +
|
||||
"\x14nvlink_replay_errors\x18\x17 \x01(\x03R\x12nvlinkReplayErrors\x124\n" +
|
||||
"\x16nvlink_recovery_errors\x18\x18 \x01(\x03R\x14nvlinkRecoveryErrors\x12\x1d\n" +
|
||||
"\n" +
|
||||
"xid_errors\x18\x19 \x01(\x03R\txidErrors\x12\x19\n" +
|
||||
"\blast_xid\x18\x1a \x01(\x05R\alastXid\x12\"\n" +
|
||||
"\rlast_xid_text\x18\x1b \x01(\tR\vlastXidText\x12\"\n" +
|
||||
"\rlast_xid_unix\x18\x1c \x01(\x03R\vlastXidUnix\"m\n" +
|
||||
"\tShardLoad\x12\x19\n" +
|
||||
"\bshard_id\x18\x01 \x01(\tR\ashardId\x12\x1b\n" +
|
||||
"\tcpu_cores\x18\x02 \x01(\x01R\bcpuCores\x12(\n" +
|
||||
@@ -4042,7 +4275,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x06status\x18\x03 \x01(\tR\x06status\x12!\n" +
|
||||
"\fcluster_uuid\x18\x04 \x01(\tR\vclusterUuid\x12\x1b\n" +
|
||||
"\tclique_id\x18\x05 \x01(\tR\bcliqueId\x12\x1c\n" +
|
||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xf6\x06\n" +
|
||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xa9\a\n" +
|
||||
"\tHeartbeat\x12\x17\n" +
|
||||
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12+\n" +
|
||||
"\x12gpu_memory_used_mb\x18\x02 \x03(\x05R\x0fgpuMemoryUsedMb\x12!\n" +
|
||||
@@ -4063,16 +4296,18 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"gpu_fabric\x18\x10 \x03(\v2\x18.llmgw.v1.GPUFabricStateR\tgpuFabric\x12\"\n" +
|
||||
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12\x1b\n" +
|
||||
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x1aA\n" +
|
||||
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x121\n" +
|
||||
"\x15gpu_foreign_memory_mb\x18\x13 \x03(\x05R\x12gpuForeignMemoryMb\x1aA\n" +
|
||||
"\x13EngineVersionsEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
||||
"\tProvision\x12.\n" +
|
||||
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\">\n" +
|
||||
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\"X\n" +
|
||||
"\n" +
|
||||
"EngineWant\x12\x16\n" +
|
||||
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\"\x8b\x01\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
|
||||
"\avariant\x18\x03 \x01(\tR\avariant\"\x8b\x01\n" +
|
||||
"\x0fProvisionStatus\x12\x16\n" +
|
||||
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
|
||||
|
||||
@@ -304,6 +304,35 @@ message GPUSensor {
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
repeated string throttle = 6;
|
||||
// How busy the card is, sampled with the rest, rather than read once at
|
||||
// registration. -1 is not reported by the driver. memory_total_mb is set
|
||||
// by every agent that sends these, so zero there is an older agent.
|
||||
double utilization_percent = 7;
|
||||
double memory_utilization_percent = 8;
|
||||
double memory_used_mb = 9;
|
||||
double memory_total_mb = 10;
|
||||
double sm_clock_mhz = 11;
|
||||
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
|
||||
double ecc_uncorrected = 12;
|
||||
// What wears the card out, read every minute; has_wear says these are
|
||||
// present. Counts of -1 are not reported for this card.
|
||||
bool has_wear = 13;
|
||||
int64 pcie_replays = 14;
|
||||
int64 retired_pages_sbe = 15;
|
||||
int64 retired_pages_dbe = 16;
|
||||
bool retirement_pending = 17;
|
||||
int64 remapped_rows_correctable = 18;
|
||||
int64 remapped_rows_uncorrectable = 19;
|
||||
bool remap_pending = 20;
|
||||
bool remap_failure = 21;
|
||||
int64 nvlink_crc_errors = 22;
|
||||
int64 nvlink_replay_errors = 23;
|
||||
int64 nvlink_recovery_errors = 24;
|
||||
// xid_errors are the driver's error events since the agent started.
|
||||
int64 xid_errors = 25;
|
||||
int32 last_xid = 26;
|
||||
string last_xid_text = 27;
|
||||
int64 last_xid_unix = 28;
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
@@ -445,6 +474,15 @@ message Heartbeat {
|
||||
IMEX imex = 17;
|
||||
// cpu_count as in Register, for a VM given more CPUs while it runs.
|
||||
int32 cpu_count = 18;
|
||||
|
||||
// gpu_foreign_memory_mb, indexed by GPU, is what processes that are not
|
||||
// this agent's shards hold on each card, read afresh every heartbeat. What
|
||||
// keeps a card out of placement for another tenant is decided from this
|
||||
// rather than from registration: a card that had something else on it when
|
||||
// the agent started is not taken forever, and one that gains a tenant later
|
||||
// is not handed out. Empty from an agent that does not measure it, or from
|
||||
// one that could not this time.
|
||||
repeated int32 gpu_foreign_memory_mb = 19;
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
@@ -465,6 +503,9 @@ message Provision {
|
||||
message EngineWant {
|
||||
string engine = 1; // "llamacpp"
|
||||
string version = 2; // "b10991"
|
||||
// variant is the build to install, when one is chosen for the node rather
|
||||
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
|
||||
string variant = 3;
|
||||
}
|
||||
|
||||
// ProvisionStatus is where one engine's installation stands on a node.
|
||||
|
||||
@@ -487,6 +487,427 @@
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "GPUs",
|
||||
"id": 14,
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 30,
|
||||
"w": 24,
|
||||
"h": 1
|
||||
},
|
||||
"collapsed": false,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU busy, by card",
|
||||
"description": "Share of each sample period a kernel was running, sampled every heartbeat.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 31,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 15,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_utilization_percent",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU memory in use, by card",
|
||||
"description": "Framebuffer in use. Engines reserve most of it at start.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 31,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 16,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_memory_used_bytes / llmgw_gpu_memory_total_bytes",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU memory bandwidth busy, by card",
|
||||
"description": "Decode is bound by memory: high here with compute low is a card waiting on memory.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 38,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 17,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_memory_utilization_percent",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU temperature",
|
||||
"description": "Throttling usually starts in the high 80s.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 38,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 18,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_temperature_celsius",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "celsius"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU power against its limit",
|
||||
"description": "1 is the enforced limit; a card held there is power-capped.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 45,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 19,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1.1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU clock",
|
||||
"description": "A clock held low under load is the card throttled; see the next panel for why.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 45,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 20,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_sm_clock_mhz",
|
||||
"legendFormat": "{{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "rotmhz"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "What holds GPUs back",
|
||||
"description": "Cards throttled now, by reason, and uncorrected ECC errors since the driver loaded.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 52,
|
||||
"w": 24,
|
||||
"h": 7
|
||||
},
|
||||
"id": 21,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (reason) (llmgw_gpu_throttled)",
|
||||
"legendFormat": "throttled: {{reason}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum by (node, gpu) (llmgw_gpu_ecc_errors)",
|
||||
"legendFormat": "ECC errors {{node}} gpu {{gpu}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Output tokens/s per GPU, by model",
|
||||
"description": "What each model gets out of each card it holds.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 59,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 22,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) / count by (model) (llmgw_gpu_assigned)",
|
||||
"legendFormat": "{{model}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Output tokens per watt-hour, by model",
|
||||
"description": "Tokens for the energy the model's cards draw.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 59,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 23,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) * 3600 / sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)",
|
||||
"legendFormat": "{{model}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU wear",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"id": 24,
|
||||
"description": "What wears cards out: XID errors in the last 15 minutes, memory rows remapped, PCIe retries an hour. Anything above zero here is worth a look on that card's node page.",
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 66,
|
||||
"w": 24,
|
||||
"h": 7
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_xid_errors[15m]) > 0",
|
||||
"legendFormat": "XID {{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum by (node, gpu) (llmgw_gpu_remapped_rows) > 0",
|
||||
"legendFormat": "rows remapped {{node}} gpu {{gpu}}",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_pcie_replays[1h]) > 0",
|
||||
"legendFormat": "PCIe retries/h {{node}} gpu {{gpu}}",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_nvlink_errors[1h]) > 0",
|
||||
"legendFormat": "NVLink {{kind}}/h {{node}} gpu {{gpu}}",
|
||||
"refId": "D"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -200,6 +200,94 @@ groups:
|
||||
is skipped by placement rather than failing, so this is quiet until
|
||||
somebody wonders why a model will not land there.
|
||||
|
||||
- name: llmgw-gpus
|
||||
rules:
|
||||
- alert: GPUThermalThrottling
|
||||
expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})"
|
||||
description: >-
|
||||
Ten minutes held back by temperature: every replica on the card is
|
||||
slower than its measurements say. Usually airflow, a fan, or the
|
||||
room — the node page shows the card's temperature over time.
|
||||
|
||||
- alert: GPUUncorrectedECCErrors
|
||||
expr: llmgw_gpu_ecc_errors > 0
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors"
|
||||
description: >-
|
||||
Uncorrected memory errors since the driver loaded: what runs on the
|
||||
card may be computing on corrupted memory. Drain the node
|
||||
(llmgw ctl drain) and have the card looked at.
|
||||
|
||||
- alert: GPUXidError
|
||||
expr: increase(llmgw_gpu_xid_errors[15m]) > 0
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
|
||||
description: >-
|
||||
The driver reported an error on the card (its code is
|
||||
llmgw_gpu_last_xid, and the node page says it in words). 79 is a
|
||||
card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
|
||||
94/95 contained and uncontained ECC errors: those need the card
|
||||
looked at. Others are often an engine's own fault, and pass.
|
||||
|
||||
- alert: GPUOutOfSpareRows
|
||||
expr: llmgw_gpu_remap_failure == 1
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
|
||||
description: >-
|
||||
The card cannot remap another failing row. Drain the node and
|
||||
have the card replaced.
|
||||
|
||||
- alert: GPUNeedsReset
|
||||
expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
|
||||
description: >-
|
||||
Rows remapped or pages retired take effect only after the card is
|
||||
reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
|
||||
it back.
|
||||
|
||||
- alert: GPUPCIeRetriesClimbing
|
||||
expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
|
||||
description: >-
|
||||
The card's link to the host is retrying. A riser, the slot or a
|
||||
cable is the usual cause, and transfers to the card slow down.
|
||||
|
||||
- alert: GPUNVLinkErrorsClimbing
|
||||
expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
|
||||
description: >-
|
||||
Errors on the card's NVLink links: tensor-parallel replicas on it
|
||||
slow down, and a link that keeps failing takes a replica with it.
|
||||
|
||||
- alert: GPUHot
|
||||
expr: llmgw_gpu_temperature_celsius > 87
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C"
|
||||
description: >-
|
||||
Ten minutes above 87 °C, where most data-centre cards begin to
|
||||
throttle.
|
||||
|
||||
- name: monitoring-host
|
||||
rules:
|
||||
- alert: MonitoringDiskFilling
|
||||
|
||||
+68
-10
@@ -367,7 +367,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
|
||||
| Engine | On | How |
|
||||
|---|---|---|
|
||||
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
|
||||
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
|
||||
@@ -376,8 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
|
||||
vLLM is built for CUDA 13. A V100 cannot run it;
|
||||
- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
|
||||
the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
|
||||
CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
|
||||
publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
|
||||
(7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
|
||||
**vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
|
||||
kernels — and the cluster page shows it running that version; run `llmgw
|
||||
ctl engine-check vllm` there once, since llmgw's flags are checked against
|
||||
the current one. Pascal and older run no vLLM. Whatever is chosen, the
|
||||
install ends by computing something on the GPU; a build that cannot is
|
||||
passed over for the next;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -394,8 +403,51 @@ engines:
|
||||
```
|
||||
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself instead, set its version to `""` and name its path in
|
||||
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
|
||||
you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
|
||||
build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
|
||||
*CUDA 12.9*, or *none* — or from the command line:
|
||||
|
||||
```
|
||||
llmgw ctl engine-build gpu07 vllm cu129
|
||||
```
|
||||
|
||||
The choice is kept by the control plane and the node is told at once; it
|
||||
installs the build in the background, and the column shows how that stands.
|
||||
|
||||
#### Different GPUs, different engines
|
||||
|
||||
A cluster of two GPU generations may need two builds of an engine, or one on
|
||||
only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
|
||||
id, or by labels, every one given having to match — and lays engines over the
|
||||
cluster's for them, in order:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: a10g # pin a class to a version you have tested
|
||||
engines: { vllm: "0.29.0" }
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
|
||||
A node told nothing it can run of an engine joins without it: `llmgw doctor`
|
||||
and `node add` warn, and the node serves what it does run. To keep an engine
|
||||
off a node for good, whatever the control plane says, add the node with
|
||||
`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
|
||||
into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
|
||||
than `--ignore-checks`, which passes over every other problem as well.
|
||||
|
||||
A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
|
||||
asks it which devices it sees, and a replica placed on GPUs is not ready until
|
||||
its process holds memory on each of them. A build missing a CUDA library fails
|
||||
there, naming the library, rather than serving on the CPU.
|
||||
|
||||
After changing a version, check it still takes the flags llmgw builds for it,
|
||||
which drift between releases:
|
||||
@@ -1032,7 +1084,12 @@ A replica may come back marked **`ready (stale profile)`**. That means it is
|
||||
running a profile that has since been edited — the reconciler leaves a serving
|
||||
replica alone on purpose, because churning one costs more than the difference.
|
||||
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
|
||||
the new one before stopping the old.
|
||||
the new one before stopping the old. Where the old one is all that stands in
|
||||
the new one's way — a single GPU — it is stopped first, at once, and the model
|
||||
is unserved while the new one loads; where the new profile fits nowhere even
|
||||
then, the restart is refused and the old one keeps serving, which `ctl
|
||||
replicas` shows as `restart refused:` with the reason. `--in-place` asks for
|
||||
stop-then-start whatever room there is.
|
||||
|
||||
### Signed releases
|
||||
|
||||
@@ -1214,11 +1271,12 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
can see each other, seeding between them is the obvious next thing to build.
|
||||
- **A budget without a price never binds.** Set `default_price` or `pricing`
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
displays rather than something that stops anything. The Limits tab says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
|
||||
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
|
||||
install a vLLM built for your CUDA yourself.
|
||||
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
|
||||
no build from 0.20 on has Volta kernels, and the node picks the newest that
|
||||
has. llmgw's flags are checked against the current version, so run `llmgw
|
||||
ctl engine-check vllm` on a V100 node once.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
+16
-6
@@ -191,7 +191,8 @@ project or person, the teams and projects with their limits and spend, and every
|
||||
member. Only what was spent in that organization's projects is on it; what its
|
||||
people do elsewhere is not. The page is read-only — a manager is not an
|
||||
administrator of the cluster, and limits and memberships stay with those who
|
||||
are.
|
||||
are. An administrator following `/org` lands on the same organization's panel
|
||||
on the people page instead, where all of that can be changed.
|
||||
|
||||
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
|
||||
will no longer move a team or project that already exists under another parent;
|
||||
@@ -259,7 +260,7 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
|
||||
A model's canonical name and any of its aliases name the same thing, so either
|
||||
form works.
|
||||
|
||||
Budgets are given in whole units, as the quotas page shows them, and what a unit
|
||||
Budgets are given in whole units, as the Limits tab shows them, and what a unit
|
||||
means is yours to decide. Self-hosted models have no per-token price, so it can
|
||||
be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
|
||||
a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
|
||||
@@ -409,10 +410,19 @@ page exists.
|
||||
opening onto its members and its keys. From there an admin approves people
|
||||
(into a project or on their own), adds them to and takes them out of
|
||||
organizations, teams and projects, suspends and removes them, revokes keys, and
|
||||
adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
|
||||
sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
|
||||
bottom. The page works as plain forms; with scripts on, a change updates it in
|
||||
place and the projects you had open stay open.
|
||||
adds, renames, moves and removes organizations, teams and projects. A
|
||||
searchable list of every account is at the bottom. Choosing any organization,
|
||||
team, project or person opens a panel beside the tree with its limits, its
|
||||
capacity quota and its use over the last week, and its limits can be changed
|
||||
there.
|
||||
|
||||
The same page has two more tabs, each at the address it had as a page of its
|
||||
own: **Limits** (`/admin/quotas`) is every limit in the tree at once — per
|
||||
branch, per member of any part of it, and per person — with the capacity quotas;
|
||||
**Usage** (`/admin/usage`) is the cluster's use over time by organization, team,
|
||||
project or person, and the GPU time each owner held. The page works as plain
|
||||
forms; with scripts on, a change updates it in place and the projects you had
|
||||
open stay open.
|
||||
|
||||
Three things are deliberately not there:
|
||||
|
||||
|
||||
@@ -81,6 +81,10 @@ rather than repeated.
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
|
||||
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
||||
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
@@ -183,6 +187,7 @@ rather than repeated.
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
@@ -645,6 +650,32 @@ called. Empty means vLLM's names.
|
||||
|
||||
`int`
|
||||
|
||||
## EngineOverride
|
||||
|
||||
### `gpu_class`
|
||||
|
||||
`string`
|
||||
|
||||
GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes picks nodes by id.
|
||||
|
||||
### `labels`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Labels picks nodes carrying all of these.
|
||||
|
||||
### `engines`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Engines are laid over the cluster's: a version, or "" for none.
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
@@ -1412,6 +1443,16 @@ never trusted to already have what somebody once put there.
|
||||
Unset keeps DefaultEngines. A version replaces the default; an empty
|
||||
one leaves that engine to be installed by hand.
|
||||
|
||||
### `engine_overrides`
|
||||
|
||||
`[]EngineOverride`
|
||||
|
||||
EngineOverrides give some nodes different engines from the rest: a
|
||||
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
engines over the cluster's, in order, later ones last; an empty version
|
||||
removes that engine from the nodes it picks.
|
||||
|
||||
### `backup_dir`
|
||||
|
||||
`string`
|
||||
|
||||
+66
-5
@@ -104,6 +104,8 @@ type Agent struct {
|
||||
// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
|
||||
// from nvidia-smi.
|
||||
sensors *discovery.SensorSampler
|
||||
// wear reads what wears a GPU out: remapped rows, PCIe retries, XIDs.
|
||||
wear *discovery.HealthSampler
|
||||
|
||||
// fetchWatchers are the shards waiting on each source, so progress on one
|
||||
// download reaches every shard that asked for it. A fetch is single-flight
|
||||
@@ -150,6 +152,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
provKick: make(chan struct{}, 1),
|
||||
load: discovery.NewLoadSampler(),
|
||||
sensors: discovery.NewSensorSampler(),
|
||||
wear: discovery.NewHealthSampler(),
|
||||
}
|
||||
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
|
||||
return a, nil
|
||||
@@ -314,6 +317,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
return
|
||||
}
|
||||
// An engine started by an agent older than the GPU check may be one
|
||||
// serving a GPU placement on the CPU. Stopped, so its replacement starts
|
||||
// through the check and fails there in words, if the build still sees
|
||||
// no GPU — rather than going on unnoticed after the upgrade that could
|
||||
// have caught it.
|
||||
if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
|
||||
a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
_ = a.sup.Stop(ctx, o.ShardID)
|
||||
return
|
||||
}
|
||||
a.log.Info("adopted engine is serving",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID)
|
||||
}
|
||||
@@ -450,7 +464,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
// vLLM accepted a vLLM shard and failed it minutes later.
|
||||
reg.EngineVersions = a.engines(parent)
|
||||
reg.EnginesProbed = true
|
||||
reg.InstallableEngines = provision.Installable()
|
||||
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
|
||||
reg.Features = []string{FeatureFabricTest}
|
||||
reg.RunningShards = a.runningShards()
|
||||
a.mu.Lock()
|
||||
@@ -577,8 +591,24 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
|
||||
}
|
||||
a.mu.Unlock()
|
||||
|
||||
for _, g := range inv.GPUs {
|
||||
hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
|
||||
// Measured now, not carried from registration: a card is shared with
|
||||
// whatever else runs on the machine, and that comes and goes while the
|
||||
// agent runs. The registration figure stays only for a node where
|
||||
// nvidia-smi cannot be asked.
|
||||
if len(inv.GPUs) > 0 {
|
||||
var shards []int
|
||||
for _, st := range a.sup.Statuses() {
|
||||
if st.PID > 0 {
|
||||
shards = append(shards, st.PID)
|
||||
}
|
||||
}
|
||||
hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = a.foreignGPUMemory(context.Background(), shards)
|
||||
}
|
||||
if len(hb.GpuMemoryUsedMb) != len(inv.GPUs) {
|
||||
hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = nil, nil
|
||||
for _, g := range inv.GPUs {
|
||||
hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
|
||||
}
|
||||
}
|
||||
|
||||
// Rescanned here rather than reused from registration. Disk is not a fact
|
||||
@@ -645,11 +675,34 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
|
||||
if a.sensors != nil {
|
||||
se := a.sensors.Sample(context.Background())
|
||||
out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
|
||||
var wear map[int]discovery.GPUWear
|
||||
if a.wear != nil && len(se.GPUs) > 0 {
|
||||
wear = map[int]discovery.GPUWear{}
|
||||
for _, w := range a.wear.Sample(context.Background()) {
|
||||
wear[w.Index] = w
|
||||
}
|
||||
}
|
||||
for _, g := range se.GPUs {
|
||||
out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
|
||||
gs := &llmgwv1.GPUSensor{
|
||||
Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
|
||||
PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
|
||||
})
|
||||
UtilizationPercent: g.UtilPercent, MemoryUtilizationPercent: g.MemUtilPercent,
|
||||
MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
|
||||
SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
|
||||
}
|
||||
if w, ok := wear[g.Index]; ok {
|
||||
gs.HasWear = true
|
||||
gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
|
||||
w.PCIeReplays, w.RetiredSBE, w.RetiredDBE, w.RetirePending
|
||||
gs.RemappedRowsCorrectable, gs.RemappedRowsUncorrectable, gs.RemapPending, gs.RemapFailure =
|
||||
w.RemappedCorrectable, w.RemappedUncorrectable, w.RemapPending, w.RemapFailure
|
||||
gs.NvlinkCrcErrors, gs.NvlinkReplayErrors, gs.NvlinkRecoveryErrors = w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery
|
||||
gs.XidErrors, gs.LastXid, gs.LastXidText = w.XIDs, int32(w.LastXID), w.LastXIDText
|
||||
if !w.LastXIDAt.IsZero() {
|
||||
gs.LastXidUnix = w.LastXIDAt.Unix()
|
||||
}
|
||||
}
|
||||
out.Gpus = append(out.Gpus, gs)
|
||||
}
|
||||
}
|
||||
return out
|
||||
@@ -931,6 +984,14 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
} else {
|
||||
err = a.sup.WaitAlive(ctx, sh)
|
||||
}
|
||||
if err == nil {
|
||||
err = a.onItsGPUs(ctx, eng, spec, sh.Status().PID)
|
||||
if err != nil {
|
||||
// Serving, but not where it was placed: stopped, so the GPU it
|
||||
// holds in the inventory is not taken by a CPU process.
|
||||
_ = a.sup.Stop(ctx, shardID)
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
report(supervisor.Failed, err.Error(), sh.Logs(20))
|
||||
return
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
func TestAnEngineDisabledOnTheNodeIsNeitherInstalledNorAdvertised(t *testing.T) {
|
||||
a := &Agent{}
|
||||
a.cfg.Engines.Disabled = []string{"vllm"}
|
||||
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{
|
||||
{Engine: "llamacpp", Version: "b1"}, {Engine: "vllm", Version: "0.30.0"},
|
||||
}})
|
||||
if len(a.provWants) != 1 || a.provWants[0].Engine != "llamacpp" {
|
||||
t.Errorf("wants %+v: vLLM is disabled here and must not be installed", a.provWants)
|
||||
}
|
||||
a.configuredVers = map[string]string{"vllm": "0.25.0"}
|
||||
a.installedVers = map[string]string{"llamacpp": "b1"}
|
||||
if got := a.engineReportLocked(); got["vllm"] != "" || got["llamacpp"] != "b1" {
|
||||
t.Errorf("advertised %v: a disabled engine found on disk is still not offered", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineNamedByHandIsNotInstalledOverIt(t *testing.T) {
|
||||
a := &Agent{reg: engines.NewRegistry(engines.Config{LlamaCpp: engines.LlamaCppConfig{Bin: "/opt/own/llama-server"}})}
|
||||
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "llamacpp", Version: "b1"}}})
|
||||
if len(a.provWants) != 0 {
|
||||
t.Errorf("wants %+v: an installed build would be used over the operator's own", a.provWants)
|
||||
}
|
||||
}
|
||||
@@ -106,9 +106,27 @@ func parseSMI(out string) ([]GPU, error) {
|
||||
return gpus, nil
|
||||
}
|
||||
|
||||
// residentMemoryToleranceMB is how much allocated memory is written off as
|
||||
// ForeignMemoryToleranceMB is how much allocated memory is written off as
|
||||
// driver overhead rather than a foreign process.
|
||||
const residentMemoryToleranceMB = 512
|
||||
const ForeignMemoryToleranceMB = 512
|
||||
|
||||
// foreignMemorySuffix ends the problem a card with another tenant on it is
|
||||
// kept out of placement for. One phrase, so that the control plane can find
|
||||
// it again and drop it when the tenant leaves.
|
||||
const foreignMemorySuffix = " MiB already allocated by another process"
|
||||
|
||||
// ForeignMemoryProblem is the problem for mb of memory held by something
|
||||
// that is not one of this agent's engines, or "" when that is too little to
|
||||
// count.
|
||||
func ForeignMemoryProblem(mb int) string {
|
||||
if mb <= ForeignMemoryToleranceMB {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf("%d%s", mb, foreignMemorySuffix)
|
||||
}
|
||||
|
||||
// IsForeignMemoryProblem reports whether a problem is ForeignMemoryProblem's.
|
||||
func IsForeignMemoryProblem(p string) bool { return strings.HasSuffix(p, foreignMemorySuffix) }
|
||||
|
||||
// assessHealth flags conditions that should keep a GPU out of placement.
|
||||
func assessHealth(g *GPU) {
|
||||
@@ -119,8 +137,12 @@ func assessHealth(g *GPU) {
|
||||
// Memory already in use means something else is on the card. Starting an
|
||||
// engine there would either fail to allocate or quietly evict the other
|
||||
// process's throughput.
|
||||
if g.MemoryUsedMB > residentMemoryToleranceMB {
|
||||
g.Health.Problems = append(g.Health.Problems,
|
||||
fmt.Sprintf("%d MiB already allocated by another process", g.MemoryUsedMB))
|
||||
//
|
||||
// Only a first reading: this runs before the agent knows which processes
|
||||
// are its own, so an engine still running from before a restart reads as
|
||||
// foreign here. Every heartbeat measures it again, without the agent's
|
||||
// own engines, and the control plane replaces this with that.
|
||||
if p := ForeignMemoryProblem(g.MemoryUsedMB); p != "" {
|
||||
g.Health.Problems = append(g.Health.Problems, p)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// GPUWear is what wears a card out or takes it down, beyond temperature:
|
||||
// read every minute, from nvidia-smi and the kernel's log. The early signs of
|
||||
// a card failing are here — a growing count of rows its memory had to remap,
|
||||
// PCIe link retries, XID errors — long before a replica on it fails.
|
||||
//
|
||||
// A count of -1 is not reported: a card without row remapping (before
|
||||
// Ampere) has no such count, which is not the same as none.
|
||||
type GPUWear struct {
|
||||
Index int
|
||||
// PCIeReplays are link-level retries on the card's PCIe link: a few are
|
||||
// noise, a count that keeps climbing is a riser, slot or cable.
|
||||
PCIeReplays int64
|
||||
// RetiredSBE and RetiredDBE are memory pages retired for single and
|
||||
// double bit errors (cards before Ampere); RetirePending is a page
|
||||
// waiting for a reset to be retired.
|
||||
RetiredSBE, RetiredDBE int64
|
||||
RetirePending bool
|
||||
// RemappedCorrectable and RemappedUncorrectable are memory rows remapped
|
||||
// (Ampere and later). RemapPending waits for a reset; RemapFailure is a
|
||||
// card out of spare rows — it needs replacing.
|
||||
RemappedCorrectable, RemappedUncorrectable int64
|
||||
RemapPending, RemapFailure bool
|
||||
// NVLink errors summed over the card's links, by kind.
|
||||
NVLinkCRC, NVLinkReplay, NVLinkRecovery int64
|
||||
// XIDs are the driver's error events for this card since the agent
|
||||
// started, and the last of them.
|
||||
XIDs int64
|
||||
LastXID int
|
||||
LastXIDText string
|
||||
LastXIDAt time.Time
|
||||
}
|
||||
|
||||
// healthEvery is how often health is read: it changes over hours, and the
|
||||
// kernel log and nvidia-smi's full report are the heavier reads.
|
||||
const healthEvery = time.Minute
|
||||
|
||||
// HealthSampler reads GPUWear, keeping the XID counts between reads.
|
||||
type HealthSampler struct {
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
at time.Time
|
||||
last []GPUWear
|
||||
since time.Time
|
||||
xids map[int]*GPUWear
|
||||
started bool
|
||||
}
|
||||
|
||||
// NewHealthSampler reads the machine's own nvidia-smi and kernel log.
|
||||
func NewHealthSampler() *HealthSampler {
|
||||
return &HealthSampler{now: time.Now, xids: map[int]*GPUWear{},
|
||||
run: func(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
return exec.CommandContext(ctx, name, args...).Output() //nolint:gosec // fixed tools
|
||||
}}
|
||||
}
|
||||
|
||||
// Sample is the cards' health, read again once healthEvery has passed. Nil
|
||||
// on a machine without nvidia-smi.
|
||||
func (h *HealthSampler) Sample(ctx context.Context) []GPUWear {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
now := h.now()
|
||||
if !h.at.IsZero() && now.Sub(h.at) < healthEvery {
|
||||
return h.last
|
||||
}
|
||||
h.at = now
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
report, err := h.run(ctx, "nvidia-smi", "-q", "-x")
|
||||
if err != nil {
|
||||
h.last = nil
|
||||
return nil
|
||||
}
|
||||
cards, bus := parseHealthReport(report)
|
||||
if out, err := h.run(ctx, "nvidia-smi", "nvlink", "-e"); err == nil {
|
||||
for i, e := range parseNVLinkErrors(string(out)) {
|
||||
if i < len(cards) {
|
||||
cards[i].NVLinkCRC, cards[i].NVLinkReplay, cards[i].NVLinkRecovery = e[0], e[1], e[2]
|
||||
}
|
||||
}
|
||||
}
|
||||
h.readXIDs(ctx, now, bus)
|
||||
for i := range cards {
|
||||
if x := h.xids[i]; x != nil {
|
||||
cards[i].XIDs, cards[i].LastXID, cards[i].LastXIDText, cards[i].LastXIDAt = x.XIDs, x.LastXID, x.LastXIDText, x.LastXIDAt
|
||||
}
|
||||
}
|
||||
h.last = cards
|
||||
return cards
|
||||
}
|
||||
|
||||
// readXIDs adds the XID events logged since the last read. The first read
|
||||
// starts from now: what happened before the agent started was before, and
|
||||
// counting the whole log would raise every alert on every restart.
|
||||
func (h *HealthSampler) readXIDs(ctx context.Context, now time.Time, bus map[string]int) {
|
||||
if !h.started {
|
||||
h.started, h.since = true, now
|
||||
return
|
||||
}
|
||||
since := h.since
|
||||
h.since = now
|
||||
out, err := h.run(ctx, "journalctl", "-k", "--no-pager", "-o", "short-unix",
|
||||
"--since", "@"+strconv.FormatInt(since.Unix(), 10), "-g", "NVRM: Xid")
|
||||
if err != nil {
|
||||
// No journal, or nothing matched (journalctl exits 1 for that).
|
||||
return
|
||||
}
|
||||
for _, e := range parseXIDs(string(out)) {
|
||||
i, ok := bus[e.bus]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
x := h.xids[i]
|
||||
if x == nil {
|
||||
x = &GPUWear{Index: i}
|
||||
h.xids[i] = x
|
||||
}
|
||||
x.XIDs++
|
||||
x.LastXID, x.LastXIDText = e.code, e.text
|
||||
x.LastXIDAt = now
|
||||
if e.at > 0 {
|
||||
x.LastXIDAt = time.Unix(int64(e.at), 0)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// healthLog is the part of nvidia-smi's XML report health is read from.
|
||||
type healthLog struct {
|
||||
GPUs []struct {
|
||||
ID string `xml:"id,attr"`
|
||||
PCI struct {
|
||||
Replay string `xml:"replay_counter"`
|
||||
} `xml:"pci"`
|
||||
Retired struct {
|
||||
SBE string `xml:"multiple_single_bit_retirement>retired_count"`
|
||||
DBE string `xml:"double_bit_retirement>retired_count"`
|
||||
Pending string `xml:"pending_retirement"`
|
||||
// Older drivers name it pending_blacklist.
|
||||
Blacklist string `xml:"pending_blacklist"`
|
||||
} `xml:"retired_pages"`
|
||||
Remapped struct {
|
||||
Corr string `xml:"remapped_row_corr"`
|
||||
Unc string `xml:"remapped_row_unc"`
|
||||
Pending string `xml:"remapped_row_pending"`
|
||||
Failure string `xml:"remapped_row_failure"`
|
||||
} `xml:"remapped_rows"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseHealthReport reads each card's health, in index order, and maps each
|
||||
// card's PCI bus ("3b:00", as the kernel names it) to its index.
|
||||
func parseHealthReport(report []byte) ([]GPUWear, map[string]int) {
|
||||
var log healthLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
bus := map[string]int{}
|
||||
out := make([]GPUWear, 0, len(log.GPUs))
|
||||
for i, g := range log.GPUs {
|
||||
bus[busOf(g.ID)] = i
|
||||
out = append(out, GPUWear{
|
||||
Index: i,
|
||||
PCIeReplays: count(g.PCI.Replay),
|
||||
RetiredSBE: count(g.Retired.SBE),
|
||||
RetiredDBE: count(g.Retired.DBE),
|
||||
RetirePending: yes(g.Retired.Pending) || yes(g.Retired.Blacklist),
|
||||
RemappedCorrectable: count(g.Remapped.Corr),
|
||||
RemappedUncorrectable: count(g.Remapped.Unc),
|
||||
RemapPending: yes(g.Remapped.Pending),
|
||||
RemapFailure: yes(g.Remapped.Failure),
|
||||
})
|
||||
}
|
||||
return out, bus
|
||||
}
|
||||
|
||||
// busOf is a PCI address as the kernel's XID lines carry it: nvidia-smi
|
||||
// says "00000000:3B:00.0", the kernel "0000:3b:00".
|
||||
func busOf(id string) string {
|
||||
id = strings.ToLower(strings.TrimSpace(id))
|
||||
if i := strings.LastIndex(id, "."); i >= 0 {
|
||||
id = id[:i]
|
||||
}
|
||||
parts := strings.Split(id, ":")
|
||||
if len(parts) >= 2 {
|
||||
return parts[len(parts)-2] + ":" + parts[len(parts)-1]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func count(s string) int64 {
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 64)
|
||||
if err != nil {
|
||||
return -1
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func yes(s string) bool { return strings.EqualFold(strings.TrimSpace(s), "yes") }
|
||||
|
||||
// nvlinkError matches a line of `nvidia-smi nvlink -e`, whose wording varies
|
||||
// by driver: "Link 0: Replay Errors: 0", "Link 0: CRC Errors: 0",
|
||||
// "Link 3: Data CRC Errors: 2".
|
||||
var (
|
||||
nvlinkGPU = regexp.MustCompile(`^GPU (\d+):`)
|
||||
nvlinkError = regexp.MustCompile(`Link \d+:\s*(.*?)\s*Errors?\s*:\s*(\d+)`)
|
||||
)
|
||||
|
||||
// parseNVLinkErrors sums each card's link errors by kind: CRC, replay,
|
||||
// recovery.
|
||||
func parseNVLinkErrors(out string) map[int][3]int64 {
|
||||
res := map[int][3]int64{}
|
||||
gpu := -1
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if m := nvlinkGPU.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
|
||||
gpu, _ = strconv.Atoi(m[1])
|
||||
continue
|
||||
}
|
||||
m := nvlinkError.FindStringSubmatch(line)
|
||||
if m == nil || gpu < 0 {
|
||||
continue
|
||||
}
|
||||
n, _ := strconv.ParseInt(m[2], 10, 64)
|
||||
e := res[gpu]
|
||||
kind := strings.ToLower(m[1])
|
||||
switch {
|
||||
case strings.Contains(kind, "crc"):
|
||||
e[0] += n
|
||||
case strings.Contains(kind, "replay"):
|
||||
e[1] += n
|
||||
case strings.Contains(kind, "recovery"):
|
||||
e[2] += n
|
||||
}
|
||||
res[gpu] = e
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
type xidEvent struct {
|
||||
bus string
|
||||
code int
|
||||
text string
|
||||
at float64
|
||||
}
|
||||
|
||||
// xidLine matches the driver's report of an XID:
|
||||
// "NVRM: Xid (PCI:0000:3b:00): 79, pid=..., GPU has fallen off the bus."
|
||||
var xidLine = regexp.MustCompile(`NVRM: Xid \(PCI:([0-9a-fA-F:.]+)\): (\d+),\s*(.*)$`)
|
||||
|
||||
// parseXIDs reads journalctl's short-unix lines: a timestamp, then the
|
||||
// message.
|
||||
func parseXIDs(out string) []xidEvent {
|
||||
var evs []xidEvent
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
m := xidLine.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
code, _ := strconv.Atoi(m[2])
|
||||
e := xidEvent{bus: busOf(m[1]), code: code, text: strings.TrimSpace(m[3])}
|
||||
if f := strings.Fields(line); len(f) > 0 {
|
||||
e.at, _ = strconv.ParseFloat(f[0], 64)
|
||||
}
|
||||
evs = append(evs, e)
|
||||
}
|
||||
return evs
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const healthXML = `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<gpu id="00000000:3B:00.0">
|
||||
<pci><replay_counter>0</replay_counter></pci>
|
||||
<retired_pages>
|
||||
<multiple_single_bit_retirement><retired_count>N/A</retired_count></multiple_single_bit_retirement>
|
||||
<double_bit_retirement><retired_count>N/A</retired_count></double_bit_retirement>
|
||||
<pending_retirement>N/A</pending_retirement>
|
||||
</retired_pages>
|
||||
<remapped_rows>
|
||||
<remapped_row_corr>2</remapped_row_corr>
|
||||
<remapped_row_unc>1</remapped_row_unc>
|
||||
<remapped_row_pending>Yes</remapped_row_pending>
|
||||
<remapped_row_failure>No</remapped_row_failure>
|
||||
</remapped_rows>
|
||||
</gpu>
|
||||
<gpu id="00000000:5E:00.0">
|
||||
<pci><replay_counter>4021</replay_counter></pci>
|
||||
<retired_pages>
|
||||
<multiple_single_bit_retirement><retired_count>3</retired_count></multiple_single_bit_retirement>
|
||||
<double_bit_retirement><retired_count>1</retired_count></double_bit_retirement>
|
||||
<pending_blacklist>Yes</pending_blacklist>
|
||||
</retired_pages>
|
||||
<remapped_rows>N/A</remapped_rows>
|
||||
</gpu>
|
||||
</nvidia_smi_log>`
|
||||
|
||||
const nvlinkOut = `GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-aaa)
|
||||
Link 0: Replay Errors: 1
|
||||
Link 0: Recovery Errors: 0
|
||||
Link 0: CRC Errors: 5
|
||||
Link 1: Data CRC Errors: 2
|
||||
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-bbb)
|
||||
Link 0: Replay Errors: 0
|
||||
`
|
||||
|
||||
func TestAGPUsWearAndFailuresAreReadAndCountedFromWhenTheAgentStarted(t *testing.T) {
|
||||
now := time.Unix(1_800_000_000, 0)
|
||||
journal := ""
|
||||
h := &HealthSampler{now: func() time.Time { return now }, xids: map[int]*GPUWear{},
|
||||
run: func(_ context.Context, name string, args ...string) ([]byte, error) {
|
||||
switch {
|
||||
case name == "nvidia-smi" && args[0] == "-q":
|
||||
return []byte(healthXML), nil
|
||||
case name == "nvidia-smi" && args[0] == "nvlink":
|
||||
return []byte(nvlinkOut), nil
|
||||
case name == "journalctl":
|
||||
if journal == "" {
|
||||
return nil, errors.New("exit status 1")
|
||||
}
|
||||
return []byte(journal), nil
|
||||
}
|
||||
return nil, errors.New("unexpected " + name)
|
||||
}}
|
||||
got := h.Sample(t.Context())
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
a, b := got[0], got[1]
|
||||
if a.RemappedCorrectable != 2 || a.RemappedUncorrectable != 1 || !a.RemapPending || a.RemapFailure ||
|
||||
a.RetiredSBE != -1 || a.PCIeReplays != 0 || a.NVLinkCRC != 7 || a.NVLinkReplay != 1 {
|
||||
t.Errorf("card 0 %+v", a)
|
||||
}
|
||||
if b.PCIeReplays != 4021 || b.RetiredSBE != 3 || b.RetiredDBE != 1 || !b.RetirePending || b.RemappedCorrectable != -1 {
|
||||
t.Errorf("card 1 %+v", b)
|
||||
}
|
||||
|
||||
// An XID on card 1, logged after the agent started; read on the next
|
||||
// sample a minute on.
|
||||
journal = "1800000030.123456 gpu07 kernel: NVRM: Xid (PCI:0000:5e:00): 79, pid=1234, name=llama-server, GPU has fallen off the bus.\n"
|
||||
now = now.Add(61 * time.Second)
|
||||
got = h.Sample(t.Context())
|
||||
if got[1].XIDs != 1 || got[1].LastXID != 79 || !strings.Contains(got[1].LastXIDText, "fallen off the bus") ||
|
||||
got[1].LastXIDAt.Unix() != 1_800_000_030 || got[0].XIDs != 0 {
|
||||
t.Errorf("after an XID: %+v / %+v", got[0], got[1])
|
||||
}
|
||||
// Within the minute, the last reading.
|
||||
journal = ""
|
||||
now = now.Add(10 * time.Second)
|
||||
if again := h.Sample(t.Context()); again[1].XIDs != 1 {
|
||||
t.Errorf("%+v", again[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestBusIDsAreReadAsTheKernelWritesThem(t *testing.T) {
|
||||
for in, want := range map[string]string{"00000000:3B:00.0": "3b:00", "0000:3b:00": "3b:00", "0000:5e:00.0": "5e:00"} {
|
||||
if got := busOf(in); got != want {
|
||||
t.Errorf("%s: %s, want %s", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
|
||||
// nvidia-smi index: what a process actually computes on, as opposed to what it
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
|
||||
// backend did not load runs on the CPU without a word, and this is where that
|
||||
// shows — the process is on no GPU's list.
|
||||
//
|
||||
// From the full query rather than --query-compute-apps, which lists compute
|
||||
// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
|
||||
// would have read as not on its GPU at all.
|
||||
//
|
||||
// A node with no nvidia-smi returns nil and no error, as discovery does.
|
||||
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
if binary == "" {
|
||||
if _, err := exec.LookPath("nvidia-smi"); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
binary = "nvidia-smi"
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseGPUProcesses(out)
|
||||
}
|
||||
|
||||
// GPUMemory is one card's memory as nvidia-smi reports it at a moment: what
|
||||
// is allocated on it, and which processes hold how much of that.
|
||||
type GPUMemory struct {
|
||||
UsedMB int
|
||||
Procs []GPUProcess
|
||||
}
|
||||
|
||||
// GPUProcess is one process holding memory on a card.
|
||||
type GPUProcess struct {
|
||||
PID int
|
||||
UsedMB int
|
||||
}
|
||||
|
||||
// GPUMemoryByProcess reports each GPU's memory, by index, with the processes
|
||||
// holding it — from the same report as GPUProcesses, so the two never
|
||||
// disagree about who is on a card.
|
||||
//
|
||||
// A node with no nvidia-smi returns nil and no error, as discovery does.
|
||||
func GPUMemoryByProcess(ctx context.Context, binary string) ([]GPUMemory, error) {
|
||||
if binary == "" {
|
||||
if _, err := exec.LookPath("nvidia-smi"); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
binary = "nvidia-smi"
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseGPUMemory(out)
|
||||
}
|
||||
|
||||
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
|
||||
// index order, its memory and the processes on it.
|
||||
type smiLog struct {
|
||||
GPUs []struct {
|
||||
Used string `xml:"fb_memory_usage>used"`
|
||||
Processes []struct {
|
||||
PID string `xml:"pid"`
|
||||
Type string `xml:"type"`
|
||||
UsedMB string `xml:"used_memory"`
|
||||
} `xml:"processes>process_info"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseGPUProcesses reads the report. GPUs are listed in index order, which
|
||||
// is nvidia-smi's own numbering — the one placement assigns by.
|
||||
func parseGPUProcesses(report []byte) (map[int][]int, error) {
|
||||
mem, err := parseGPUMemory(report)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[int][]int{}
|
||||
for i, g := range mem {
|
||||
for _, p := range g.Procs {
|
||||
out[i] = append(out[i], p.PID)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// parseGPUMemory reads the report into each GPU's memory and its holders.
|
||||
func parseGPUMemory(report []byte) ([]GPUMemory, error) {
|
||||
var log smiLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
out := make([]GPUMemory, len(log.GPUs))
|
||||
for i, g := range log.GPUs {
|
||||
out[i].UsedMB = mib(g.Used)
|
||||
for _, p := range g.Processes {
|
||||
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
|
||||
out[i].Procs = append(out[i].Procs, GPUProcess{PID: pid, UsedMB: mib(p.UsedMB)})
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// mib reads "4932 MiB"; anything else, "N/A" included, is zero.
|
||||
func mib(s string) int {
|
||||
n, _ := strconv.Atoi(strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(s), "MiB")))
|
||||
return n
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
|
||||
report := `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<driver_version>580.65.06</driver_version>
|
||||
<gpu id="00000000:17:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes></processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:65:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes>
|
||||
<process_info><pid>4100</pid><type>C</type><process_name>llama-server</process_name></process_info>
|
||||
<process_info><pid>4300</pid><type>G</type><process_name>llama-server</process_name></process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:B3:00.0"><processes>
|
||||
<process_info><pid>4200</pid><type>C+G</type></process_info>
|
||||
</processes></gpu>
|
||||
</nvidia_smi_log>`
|
||||
got, err := parseGPUProcesses([]byte(report))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A Vulkan engine is a graphics process ("G") and is on its GPU all the
|
||||
// same; a compute-only query would have missed it.
|
||||
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
|
||||
t.Error("a report that does not parse was read as no processes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGPUMemoryIsReadPerProcess(t *testing.T) {
|
||||
report := `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<gpu id="00000000:01:00.0"><product_name>Quadro RTX 6000</product_name>
|
||||
<fb_memory_usage><total>24576 MiB</total><used>14514 MiB</used><free>10062 MiB</free></fb_memory_usage>
|
||||
<processes>
|
||||
<process_info><pid>71315</pid><type>C</type><used_memory>4932 MiB</used_memory></process_info>
|
||||
<process_info><pid>4100</pid><type>C</type><used_memory>9500 MiB</used_memory></process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:02:00.0">
|
||||
<fb_memory_usage><used>N/A</used></fb_memory_usage>
|
||||
<processes></processes>
|
||||
</gpu>
|
||||
</nvidia_smi_log>`
|
||||
got, err := parseGPUMemory([]byte(report))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []GPUMemory{
|
||||
{UsedMB: 14514, Procs: []GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 4100, UsedMB: 9500}}},
|
||||
{},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestForeignMemoryBelowTheToleranceIsNoProblem(t *testing.T) {
|
||||
// A CUDA context that has let go of its model still holds a few hundred
|
||||
// MiB: ComfyUI after /free held 324.
|
||||
if p := ForeignMemoryProblem(330); p != "" {
|
||||
t.Errorf("330 MiB is a problem: %q", p)
|
||||
}
|
||||
p := ForeignMemoryProblem(5014)
|
||||
if !IsForeignMemoryProblem(p) || IsForeignMemoryProblem("2 uncorrected ECC errors") {
|
||||
t.Errorf("%q is not recognised as the foreign memory problem, or ECC is", p)
|
||||
}
|
||||
}
|
||||
@@ -41,6 +41,14 @@ type GPUSensor struct {
|
||||
// is running free. Idle and application clock settings are left out;
|
||||
// they are not the card being held back.
|
||||
Throttle []string
|
||||
// How busy it is, read with the rest every sample. Discovery reads the
|
||||
// same once, at start, which made a busy card look as idle as it was
|
||||
// when the agent came up. -1 is not reported by the driver; zero
|
||||
// MemoryTotalMB is a sample without these at all.
|
||||
UtilPercent, MemUtilPercent float64
|
||||
MemoryUsedMB, MemoryTotalMB float64
|
||||
SMClockMHz float64
|
||||
ECCUncorrected float64
|
||||
}
|
||||
|
||||
// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
|
||||
@@ -187,13 +195,33 @@ var throttleBits = []struct {
|
||||
// gpus asks nvidia-smi for each card's heat and power. The memory temperature
|
||||
// is not known to every driver, and a field nvidia-smi does not know fails the
|
||||
// whole query, so it is asked for first and dropped on failure.
|
||||
// gpuFields are what each sample asks nvidia-smi for, in order. The busy
|
||||
// figures follow the throttle reasons; the memory temperature is last, being
|
||||
// the one field a driver may reject outright.
|
||||
const (
|
||||
gpuBase = "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active"
|
||||
gpuBusy = ",utilization.gpu,utilization.memory,memory.used,memory.total,clocks.sm,ecc.errors.uncorrected.volatile.total"
|
||||
gpuHBMTC = ",temperature.memory"
|
||||
)
|
||||
|
||||
func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
|
||||
withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
|
||||
out, err := s.smi(ctx, withMem)
|
||||
if err != nil {
|
||||
if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
|
||||
return nil
|
||||
// Most fields first, fewer on a driver that refuses one: a query naming
|
||||
// a field the driver does not know fails as a whole.
|
||||
var out []byte
|
||||
var busy bool
|
||||
for _, q := range []struct {
|
||||
fields string
|
||||
busy bool
|
||||
}{{gpuBase + gpuBusy + gpuHBMTC, true}, {gpuBase + gpuBusy, true}, {gpuBase + gpuHBMTC, false}, {gpuBase, false}} {
|
||||
var err error
|
||||
if out, err = s.smi(ctx, q.fields); err == nil {
|
||||
busy = q.busy
|
||||
break
|
||||
}
|
||||
out = nil
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
var gs []GPUSensor
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
@@ -216,14 +244,31 @@ func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(f) > 5 {
|
||||
g.MemoryTempC = num(f[5])
|
||||
rest := f[5:]
|
||||
if busy && len(rest) >= 6 {
|
||||
g.UtilPercent, g.MemUtilPercent = numOrUnknown(rest[0]), numOrUnknown(rest[1])
|
||||
g.MemoryUsedMB, g.MemoryTotalMB = num(rest[2]), num(rest[3])
|
||||
g.SMClockMHz, g.ECCUncorrected = num(rest[4]), numOrUnknown(rest[5])
|
||||
rest = rest[6:]
|
||||
}
|
||||
if len(rest) > 0 {
|
||||
g.MemoryTempC = num(rest[0])
|
||||
}
|
||||
gs = append(gs, g)
|
||||
}
|
||||
return gs
|
||||
}
|
||||
|
||||
// numOrUnknown is a figure, or -1 where nvidia-smi says it has none: a card
|
||||
// without ECC has no error count, which is not the same as none.
|
||||
func numOrUnknown(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil || v < 0 {
|
||||
return -1
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
|
||||
func num(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
|
||||
@@ -167,3 +167,44 @@ func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T
|
||||
t.Errorf("%+v", got.GPUs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachSampleSaysHowBusyEachGPUIs(t *testing.T) {
|
||||
// What the card is doing now, every sample: discovery read it once, at
|
||||
// start, and a card busy since looked as idle as it was then.
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
if !strings.Contains(fields, "utilization.gpu") {
|
||||
return nil, errors.New("asked without how busy the cards are")
|
||||
}
|
||||
return []byte(
|
||||
"0, 71, 612.45, 700.00, 0x0000000000000000, 97, 64, 71234, 81559, 1980, 0, 84\n" +
|
||||
"1, 30, 70.10, 700.00, 0x0000000000000001, 0, 0, 512, 81559, 345, [N/A], [N/A]\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
g0, g1 := got[0], got[1]
|
||||
if g0.UtilPercent != 97 || g0.MemUtilPercent != 64 || g0.MemoryUsedMB != 71234 || g0.MemoryTotalMB != 81559 ||
|
||||
g0.SMClockMHz != 1980 || g0.ECCUncorrected != 0 || g0.MemoryTempC != 84 {
|
||||
t.Errorf("card 0 %+v", g0)
|
||||
}
|
||||
// Idle is zero; a count the card does not keep is unknown, not zero.
|
||||
if g1.UtilPercent != 0 || g1.ECCUncorrected != -1 || g1.MemoryTempC != 0 {
|
||||
t.Errorf("card 1 %+v", g1)
|
||||
}
|
||||
}
|
||||
|
||||
func TestADriverThatDoesNotKnowTheBusyFieldsStillReportsHeatAndPower(t *testing.T) {
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
if strings.Contains(fields, "utilization.gpu") {
|
||||
return nil, errors.New(`Field "ecc.errors.uncorrected.volatile.total" is not a valid field to query.`)
|
||||
}
|
||||
return []byte("0, 66, 250.0, 300.0, 0x0000000000000000, 70\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
|
||||
if len(got) != 1 || got[0].TempC != 66 || got[0].MemoryTempC != 70 || got[0].MemoryTotalMB != 0 {
|
||||
t.Errorf("%+v", got)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"slices"
|
||||
@@ -146,6 +147,15 @@ type Engine interface {
|
||||
MetricsPath() string
|
||||
}
|
||||
|
||||
// CPUFallback is an engine that can come up on the CPU when it was placed on
|
||||
// GPUs, and keep serving there without an error. The agent checks such a shard
|
||||
// is on its GPUs before calling it ready.
|
||||
type CPUFallback interface {
|
||||
FallsBackToCPU() bool
|
||||
// Devices lists the GPUs the engine's build can see, for saying why.
|
||||
Devices(ctx context.Context) ([]string, error)
|
||||
}
|
||||
|
||||
// GPUInfo is the subset of GPU facts validation needs.
|
||||
type GPUInfo struct {
|
||||
Index int
|
||||
@@ -203,6 +213,10 @@ func (r *Registry) Names() []string {
|
||||
|
||||
// Config configures the engine adapters.
|
||||
type Config struct {
|
||||
// Disabled are engines this node never installs or advertises, whatever
|
||||
// the control plane asks: vLLM on a node of cards no build of it runs on,
|
||||
// that is meant for llama.cpp alone.
|
||||
Disabled []string `koanf:"disabled"`
|
||||
VLLM VLLMConfig `koanf:"vllm"`
|
||||
SGLang SGLangConfig `koanf:"sglang"`
|
||||
LlamaCpp LlamaCppConfig `koanf:"llamacpp"`
|
||||
@@ -232,6 +246,11 @@ func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
|
||||
func baseEnv(s ShardSpec) map[string]string {
|
||||
env := map[string]string{
|
||||
"CUDA_VISIBLE_DEVICES": joinInts(s.GPUIndices),
|
||||
// The indices above are nvidia-smi's, which counts by PCI bus. CUDA
|
||||
// counts fastest first unless told otherwise, and on a node with two
|
||||
// kinds of GPU the shard would get a different card than placement
|
||||
// gave it — one another shard may hold.
|
||||
"CUDA_DEVICE_ORDER": "PCI_BUS_ID",
|
||||
}
|
||||
|
||||
if len(s.HCAs) > 0 {
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sort"
|
||||
@@ -65,25 +67,25 @@ func (*LlamaCpp) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// Validate checks the profile against the GPUs it would run on.
|
||||
//
|
||||
// The two refusals here are about not pretending: llama.cpp has no equivalent
|
||||
// of a tensor-parallel group spanning machines, and its multi-GPU support is a
|
||||
// different mechanism (--split-mode, --tensor-split) that this adapter does not
|
||||
// drive. Accepting either would produce a replica that starts and then serves
|
||||
// from one machine while the catalog believes it has several.
|
||||
// llama.cpp spreads a model over a node's GPUs by layer (--split-mode layer),
|
||||
// so gpus_per_node is how many it gets. It has no tensor parallelism and no
|
||||
// group spanning machines, and accepting tp or nodes would produce a replica
|
||||
// that serves from fewer cards, or one machine, while the catalog believes
|
||||
// otherwise.
|
||||
func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if err := validateCommon(p, gpus); err != nil {
|
||||
return err
|
||||
}
|
||||
if p.Nodes > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
p.Name, p.Nodes)
|
||||
}
|
||||
if p.TP > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp does not do tensor parallelism; "+
|
||||
"it splits a model across local GPUs with --tensor-split, which this adapter does not drive",
|
||||
p.Name)
|
||||
if p.TP > 1 || p.PP > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp splits a model across a node's GPUs by layer, not by tp or pp; "+
|
||||
"write gpus_per_node: %d instead", p.Name, max(p.TP, 1)*max(p.PP, 1))
|
||||
}
|
||||
return nil
|
||||
// The common checks count a replica's GPUs as tp; for llama.cpp the
|
||||
// count is gpus_per_node.
|
||||
q := p
|
||||
q.TP = max(p.GPUsPerNode, 1)
|
||||
return validateCommon(q, gpus)
|
||||
}
|
||||
|
||||
// Commands returns the processes for one shard, which is always one process.
|
||||
@@ -132,6 +134,13 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
|
||||
args = append(args, "--n-gpu-layers", "999")
|
||||
}
|
||||
// Several GPUs: the layers spread across all of them, which is the only
|
||||
// split llama.cpp does without a matched interconnect. In proportion to
|
||||
// each card's free memory, which for the whole cards placement gives is
|
||||
// an even split.
|
||||
if len(s.GPUIndices) > 1 && !hasFlag(p.Args, "--split-mode", "-sm") {
|
||||
args = append(args, "--split-mode", "layer")
|
||||
}
|
||||
if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
|
||||
// What placement reserved first: it is what the node's other
|
||||
// replicas were placed around.
|
||||
@@ -159,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if s.APIKey != "" {
|
||||
env["LLAMA_API_KEY"] = s.APIKey
|
||||
}
|
||||
// The Vulkan build — what a GPU older than the CUDA builds gets — does
|
||||
// not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
|
||||
// as the driver lists them, which on NVIDIA follows the same bus order
|
||||
// as nvidia-smi; were it not to, the shard would not hold memory on its
|
||||
// own GPUs, and starting it checks exactly that.
|
||||
if len(s.GPUIndices) > 0 {
|
||||
env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
|
||||
}
|
||||
return []Command{{
|
||||
Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
|
||||
}}, nil
|
||||
@@ -234,3 +251,38 @@ func (l *LlamaCpp) bin() string {
|
||||
}
|
||||
return "llama-server"
|
||||
}
|
||||
|
||||
// FallsBackToCPU says that llama.cpp keeps running when its GPU backend does
|
||||
// not load: --n-gpu-layers asks for layers on a GPU it then has none of, and
|
||||
// the server comes up and answers on the CPU, at a tenth of the speed, with no
|
||||
// error anywhere. Its readiness on a GPU placement needs checking separately.
|
||||
func (*LlamaCpp) FallsBackToCPU() bool { return true }
|
||||
|
||||
// Devices lists the GPUs this build can compute on, as llama-server's
|
||||
// --list-devices names them ("CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)").
|
||||
// Empty, on a machine with GPUs, is a build whose GPU backend did not load:
|
||||
// the CPU build, or a CUDA build missing a library its backend links.
|
||||
func (l *LlamaCpp) Devices(ctx context.Context) ([]string, error) {
|
||||
out, err := exec.CommandContext(ctx, l.bin(), "--list-devices").CombinedOutput() //nolint:gosec // the engine's own binary
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s --list-devices: %v: %s", l.bin(), err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return parseDevices(string(out)), nil
|
||||
}
|
||||
|
||||
// parseDevices reads --list-devices: the lines after "Available devices:",
|
||||
// which says "(none)" when there are none. Log lines before it are skipped.
|
||||
func parseDevices(out string) []string {
|
||||
var devs []string
|
||||
listing := false
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
switch {
|
||||
case strings.HasPrefix(line, "Available devices"):
|
||||
listing = true
|
||||
case listing && line != "" && line != "(none)":
|
||||
devs = append(devs, line)
|
||||
}
|
||||
}
|
||||
return devs
|
||||
}
|
||||
@@ -323,3 +323,41 @@ func TestAdaptersAreLoadedAtStartAndAppliedToNoneByDefault(t *testing.T) {
|
||||
t.Error("a model with no adapters was given lora flags")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDevicesAreReadFromListDevices(t *testing.T) {
|
||||
cuda := "ggml_cuda_init: found 2 CUDA devices:\nAvailable devices:\n CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n CUDA1: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
|
||||
if got := parseDevices(cuda); len(got) != 2 || !strings.HasPrefix(got[0], "CUDA0: NVIDIA L4") {
|
||||
t.Errorf("CUDA build: %q", got)
|
||||
}
|
||||
// A CUDA build whose backend did not load says what the CPU build says.
|
||||
if got := parseDevices("Available devices:\n (none)\n"); len(got) != 0 {
|
||||
t.Errorf("no devices: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLlamaCppSpreadsAReplicaOverItsNodesGPUsByLayer(t *testing.T) {
|
||||
four := Profile{Name: "t4x4", Engine: "llamacpp", GPUClass: "t4", TP: 1, GPUsPerNode: 4, ModelPath: "/m"}
|
||||
if err := (&LlamaCpp{}).Validate(four, h100(4)); err != nil {
|
||||
t.Fatalf("four GPUs on a four-GPU node: %v", err)
|
||||
}
|
||||
if err := (&LlamaCpp{}).Validate(four, h100(2)); err == nil {
|
||||
t.Error("four GPUs were accepted on a node with two")
|
||||
}
|
||||
|
||||
s := llamaSpec([]int{0, 1, 2, 3})
|
||||
s.Profile.GPUsPerNode = 4
|
||||
cmds, err := (&LlamaCpp{}).Commands(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, _ := flagValue(cmds[0].Args, "--split-mode"); v != "layer" {
|
||||
t.Errorf("--split-mode %q over four GPUs", v)
|
||||
}
|
||||
if !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0,1,2,3") {
|
||||
t.Errorf("env %v: the replica must see all four of its GPUs", cmds[0].Env)
|
||||
}
|
||||
one, _ := (&LlamaCpp{}).Commands(llamaSpec([]int{2}))
|
||||
if hasFlag(one[0].Args, "--split-mode") {
|
||||
t.Error("a one-GPU replica was given a split mode")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
// gpuProcesses and processGroup are variables so tests can stand in for
|
||||
// nvidia-smi and /proc.
|
||||
var (
|
||||
gpuProcesses = discovery.GPUProcesses
|
||||
gpuMemory = discovery.GPUMemoryByProcess
|
||||
processGroup = pgidOf
|
||||
// gpuCheckWait is how long a shard has, once it answers, to show up on
|
||||
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
|
||||
gpuCheckWait = 15 * time.Second
|
||||
)
|
||||
|
||||
// onItsGPUs checks that a shard placed on GPUs computes on them.
|
||||
//
|
||||
// For an engine that falls back to the CPU when its GPU backend does not load,
|
||||
// "it answers" is not "it is ready": llama.cpp built for CUDA but missing
|
||||
// cuBLAS comes up on the CPU, answers at a tenth of the speed, and the GPU it
|
||||
// was given sits idle and reserved. What it computes on is what nvidia-smi
|
||||
// lists it on, so a shard none of whose processes hold memory on one of its
|
||||
// GPUs is refused, with what the engine itself can see.
|
||||
func (a *Agent) onItsGPUs(ctx context.Context, eng engines.Engine, spec engines.ShardSpec, pid int) error {
|
||||
fb, ok := eng.(engines.CPUFallback)
|
||||
if !ok || !fb.FallsBackToCPU() || len(spec.GPUIndices) == 0 || pid <= 0 {
|
||||
return nil
|
||||
}
|
||||
group := processGroup(pid)
|
||||
deadline := time.Now().Add(gpuCheckWait)
|
||||
var missing []int
|
||||
for {
|
||||
procs, err := gpuProcesses(ctx, a.cfg.Discovery.NvidiaSMI)
|
||||
if err != nil || procs == nil {
|
||||
// No nvidia-smi to ask is no evidence either way, and a node that
|
||||
// was given GPUs has one; say so and let the shard serve.
|
||||
a.log.Warn("could not check that a GPU shard runs on its GPUs", "shard", spec.ShardID, "err", err)
|
||||
return nil
|
||||
}
|
||||
missing = missing[:0]
|
||||
for _, i := range spec.GPUIndices {
|
||||
if !holds(procs[i], pid, group) {
|
||||
missing = append(missing, i)
|
||||
}
|
||||
}
|
||||
if len(missing) == 0 {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
seen := "it lists no GPU at all"
|
||||
if devs, err := fb.Devices(ctx); err != nil {
|
||||
seen = "listing its devices failed: " + err.Error()
|
||||
} else if len(devs) > 0 {
|
||||
seen = "it lists " + strings.Join(devs, "; ")
|
||||
}
|
||||
return fmt.Errorf("placed on GPU %s but computing on the CPU: the engine holds no memory there. "+
|
||||
"Its GPU backend did not load (%s). Check the build with `ldd` on its libggml-cuda.so, "+
|
||||
"or run `llmgw doctor` on this node", joinIndices(missing), seen)
|
||||
}
|
||||
|
||||
// foreignGPUMemory is what each GPU holds for anything other than this
|
||||
// agent's shards, by index, and each GPU's memory in use. Nil when nvidia-smi
|
||||
// cannot be asked, which is no reading rather than an empty card.
|
||||
//
|
||||
// Everything allocated less what the shards' processes hold, rather than the
|
||||
// sum of the other processes: a process in another container's PID namespace
|
||||
// holds memory nvidia-smi lists under no PID this node can see, and it is
|
||||
// still not memory placement can hand out.
|
||||
func (a *Agent) foreignGPUMemory(ctx context.Context, shards []int) (foreign, used []int32) {
|
||||
mem, err := gpuMemory(ctx, a.cfg.Discovery.NvidiaSMI)
|
||||
if err != nil || mem == nil {
|
||||
return nil, nil
|
||||
}
|
||||
groups := make([]int, len(shards))
|
||||
for i, pid := range shards {
|
||||
groups[i] = processGroup(pid)
|
||||
}
|
||||
ours := func(p int) bool {
|
||||
for i, pid := range shards {
|
||||
if holds([]int{p}, pid, groups[i]) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
for _, g := range mem {
|
||||
f := g.UsedMB
|
||||
for _, p := range g.Procs {
|
||||
if ours(p.PID) {
|
||||
f -= p.UsedMB
|
||||
}
|
||||
}
|
||||
foreign = append(foreign, int32(max(f, 0)))
|
||||
used = append(used, int32(g.UsedMB))
|
||||
}
|
||||
return foreign, used
|
||||
}
|
||||
|
||||
// holds reports whether any of pids is the shard's process or in its group:
|
||||
// an engine that computes in a child process is still the shard.
|
||||
func holds(pids []int, pid, group int) bool {
|
||||
for _, p := range pids {
|
||||
if p == pid || (group > 0 && processGroup(p) == group) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// pgidOf reads a process's group from /proc; zero when it cannot.
|
||||
func pgidOf(pid int) int {
|
||||
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
// The command name, in parentheses, may hold spaces; the fields after its
|
||||
// closing parenthesis are state, ppid, pgrp.
|
||||
s := string(b)
|
||||
i := strings.LastIndexByte(s, ')')
|
||||
if i < 0 {
|
||||
return 0
|
||||
}
|
||||
f := strings.Fields(s[i+1:])
|
||||
if len(f) < 3 {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.Atoi(f[2])
|
||||
return n
|
||||
}
|
||||
|
||||
func joinIndices(is []int) string {
|
||||
s := make([]string, len(is))
|
||||
for k, i := range is {
|
||||
s[k] = strconv.Itoa(i)
|
||||
}
|
||||
return strings.Join(s, ", ")
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
// fallbackEngine is an engine that comes up on the CPU when its GPU backend
|
||||
// does not load, and lists the devices it can see.
|
||||
type fallbackEngine struct {
|
||||
engines.Engine
|
||||
devices []string
|
||||
}
|
||||
|
||||
func (fallbackEngine) FallsBackToCPU() bool { return true }
|
||||
func (f fallbackEngine) Devices(context.Context) ([]string, error) {
|
||||
return f.devices, nil
|
||||
}
|
||||
|
||||
func TestAGPUShardComputingOnTheCPUIsNotReady(t *testing.T) {
|
||||
procs := map[int][]int{}
|
||||
oldProcs, oldGroup, oldWait := gpuProcesses, processGroup, gpuCheckWait
|
||||
gpuProcesses = func(context.Context, string) (map[int][]int, error) { return procs, nil }
|
||||
processGroup = func(pid int) int {
|
||||
if pid >= 500 && pid < 600 {
|
||||
return 500 // the shard's group: 500 and its children
|
||||
}
|
||||
return pid
|
||||
}
|
||||
gpuCheckWait = 0
|
||||
t.Cleanup(func() { gpuProcesses, processGroup, gpuCheckWait = oldProcs, oldGroup, oldWait })
|
||||
|
||||
a := &Agent{log: slog.Default()}
|
||||
spec := engines.ShardSpec{ShardID: "s", GPUIndices: []int{0, 1}}
|
||||
eng := fallbackEngine{}
|
||||
|
||||
// On no GPU at all: refused, saying what the engine sees.
|
||||
err := a.onItsGPUs(context.Background(), eng, spec, 500)
|
||||
if err == nil || !strings.Contains(err.Error(), "computing on the CPU") || !strings.Contains(err.Error(), "lists no GPU") {
|
||||
t.Fatalf("a shard on no GPU was let through: %v", err)
|
||||
}
|
||||
// On one of its two: refused, naming the one it is missing.
|
||||
procs[0] = []int{500}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil || !strings.Contains(err.Error(), "GPU 1 ") {
|
||||
t.Fatalf("a shard on half its GPUs: %v", err)
|
||||
}
|
||||
// Another process on GPU 1 is not the shard.
|
||||
procs[1] = []int{900}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil {
|
||||
t.Fatal("another process's memory was taken for the shard's")
|
||||
}
|
||||
// A child of the shard holding GPU 1 is the shard.
|
||||
procs[1] = []int{900, 512}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err != nil {
|
||||
t.Fatalf("a shard on both its GPUs: %v", err)
|
||||
}
|
||||
|
||||
// An engine that fails rather than falling back is not checked, nor is a
|
||||
// shard with no GPUs.
|
||||
procs = map[int][]int{}
|
||||
if err := a.onItsGPUs(context.Background(), struct{ engines.Engine }{}, spec, 500); err != nil {
|
||||
t.Errorf("checked an engine that does not fall back: %v", err)
|
||||
}
|
||||
if err := a.onItsGPUs(context.Background(), eng, engines.ShardSpec{}, 500); err != nil {
|
||||
t.Errorf("checked a CPU shard: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCheckWaitsForTheShardToShowUpOnItsGPU(t *testing.T) {
|
||||
calls := 0
|
||||
oldProcs, oldWait := gpuProcesses, gpuCheckWait
|
||||
gpuProcesses = func(context.Context, string) (map[int][]int, error) {
|
||||
calls++
|
||||
if calls < 2 {
|
||||
return map[int][]int{}, nil
|
||||
}
|
||||
return map[int][]int{0: {77}}, nil
|
||||
}
|
||||
gpuCheckWait = 5 * time.Second
|
||||
t.Cleanup(func() { gpuProcesses, gpuCheckWait = oldProcs, oldWait })
|
||||
|
||||
a := &Agent{log: slog.Default()}
|
||||
if err := a.onItsGPUs(context.Background(), fallbackEngine{}, engines.ShardSpec{GPUIndices: []int{0}}, 77); err != nil {
|
||||
t.Fatalf("nvidia-smi lagging by a second failed the shard: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestForeignGPUMemoryLeavesOutTheAgentsOwnShards(t *testing.T) {
|
||||
// cuda01: ComfyUI and one of our engines on the same card. Only ComfyUI's
|
||||
// share — and whatever no listed process accounts for — is foreign.
|
||||
oldMem, oldGroup := gpuMemory, processGroup
|
||||
gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) {
|
||||
return []discovery.GPUMemory{
|
||||
{UsedMB: 15000, Procs: []discovery.GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 512, UsedMB: 9500}}},
|
||||
{UsedMB: 300},
|
||||
}, nil
|
||||
}
|
||||
processGroup = func(pid int) int {
|
||||
if pid >= 500 && pid < 600 {
|
||||
return 500
|
||||
}
|
||||
return pid
|
||||
}
|
||||
t.Cleanup(func() { gpuMemory, processGroup = oldMem, oldGroup })
|
||||
|
||||
a := &Agent{log: slog.Default()}
|
||||
foreign, used := a.foreignGPUMemory(context.Background(), []int{500})
|
||||
if !slices.Equal(foreign, []int32{5500, 300}) || !slices.Equal(used, []int32{15000, 300}) {
|
||||
t.Errorf("foreign %v, used %v; want [5500 300] and [15000 300]", foreign, used)
|
||||
}
|
||||
|
||||
// No nvidia-smi to ask is no reading, not an empty card.
|
||||
gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) { return nil, nil }
|
||||
if foreign, _ := a.foreignGPUMemory(context.Background(), nil); foreign != nil {
|
||||
t.Errorf("no reading came back as %v", foreign)
|
||||
}
|
||||
}
|
||||
@@ -46,9 +46,30 @@ func (a *Agent) engineReportLocked() map[string]string {
|
||||
out = map[string]string{}
|
||||
}
|
||||
maps.Copy(out, a.installedVers)
|
||||
maps.DeleteFunc(out, func(e, _ string) bool { return a.disabled(e) })
|
||||
return out
|
||||
}
|
||||
|
||||
// byHand reports whether agent.yaml names this engine's binary, which opts
|
||||
// it out of being installed.
|
||||
func (a *Agent) byHand(engine string) bool {
|
||||
if a.reg == nil {
|
||||
return false
|
||||
}
|
||||
e, err := a.reg.Get(engine)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
c, ok := e.(engines.Configured)
|
||||
return ok && c.Configured()
|
||||
}
|
||||
|
||||
// disabled reports whether agent.yaml says this node never runs an engine:
|
||||
// not installed when asked, and not advertised when found.
|
||||
func (a *Agent) disabled(engine string) bool {
|
||||
return slices.Contains(a.cfg.Engines.Disabled, engine)
|
||||
}
|
||||
|
||||
// provisionEvery is how often installed engines are checked again. A library
|
||||
// removed by an unrelated upgrade is found here rather than by the next
|
||||
// replica to start.
|
||||
@@ -58,7 +79,16 @@ const provisionEvery = 10 * time.Minute
|
||||
func (a *Agent) setProvision(p *llmgwv1.Provision) {
|
||||
wants := []provision.Want{}
|
||||
for _, e := range p.GetEngines() {
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
|
||||
if a.disabled(e.GetEngine()) {
|
||||
continue // a control plane older than the node's say in it
|
||||
}
|
||||
if a.byHand(e.GetEngine()) {
|
||||
// The operator named a binary for it: a local build, a CUDA
|
||||
// runtime of their own. Installing one beside it would win —
|
||||
// the installed build is used first — and quietly replace theirs.
|
||||
continue
|
||||
}
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion(), Variant: e.GetVariant()})
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.provWants = wants
|
||||
|
||||
@@ -27,6 +27,9 @@ import (
|
||||
type Want struct {
|
||||
Engine string
|
||||
Version string
|
||||
// Variant is a build chosen for this node ("cu129"); empty leaves it to
|
||||
// the node's hardware.
|
||||
Variant string
|
||||
}
|
||||
|
||||
// The states an engine's installation can be in.
|
||||
@@ -135,7 +138,12 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
"; install it by hand and name it in the agent's configuration"
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, in.Host)
|
||||
if w.Variant != "" && !validVersion.MatchString(w.Variant) {
|
||||
st.State = Unsupported
|
||||
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
|
||||
return st
|
||||
}
|
||||
builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Unsupported, err.Error()
|
||||
var nh NotHere
|
||||
@@ -144,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
}
|
||||
return st
|
||||
}
|
||||
st.Variant = build.Variant
|
||||
dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
|
||||
|
||||
// Installed before: check it still runs, and trust nothing else about it.
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
|
||||
// Installed before: whichever candidate is, best first, checked that it
|
||||
// still runs and trusted for nothing else.
|
||||
for _, build := range builds {
|
||||
dir := in.dirFor(w, build)
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
|
||||
continue
|
||||
}
|
||||
st := st.as(w, build)
|
||||
// The packages are checked too. Removing one is exactly the kind of
|
||||
// change an unrelated upgrade makes, and ldd below would catch it only
|
||||
// as a missing library with no hint of which package held it.
|
||||
// change an unrelated upgrade makes, and ldd below would catch it
|
||||
// only as a missing library with no hint of which package held it.
|
||||
if err := in.packages(ctx, build.Packages, &st); err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
@@ -161,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
in.log().Warn("an installed engine no longer runs; installing it again",
|
||||
"engine", w.Engine, "version", w.Version, "err", err)
|
||||
"engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
|
||||
break
|
||||
}
|
||||
|
||||
if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
|
||||
st.State, st.Detail, st.Binary = Failed, err.Error(), ""
|
||||
return st
|
||||
// Each candidate in turn until one installs and is seen to work: a build
|
||||
// the table was wrong about — a driver too old for it after all, a GPU
|
||||
// it has no kernels for — costs its download, and the next is tried.
|
||||
var tried []string
|
||||
for _, build := range builds {
|
||||
st := st.as(w, build)
|
||||
if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
in.log().Warn("an engine build did not work here; trying the next",
|
||||
"engine", w.Engine, "variant", build.Variant, "err", err)
|
||||
continue
|
||||
}
|
||||
bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
|
||||
if err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
bin, version, err := in.verify(ctx, dir, build)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
st.State, st.Binary = Failed, ""
|
||||
st.Detail = strings.Join(tried, "; then ")
|
||||
if len(builds) == 1 {
|
||||
_, st.Detail, _ = strings.Cut(st.Detail, ": ")
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
return st
|
||||
}
|
||||
|
||||
// dirFor is where a build of an engine is installed: by the version it is
|
||||
// and its variant, so two builds of one version sit side by side.
|
||||
func (in *Installer) dirFor(w Want, b Build) string {
|
||||
return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
|
||||
}
|
||||
|
||||
// versionOr is the version a build is: its own, when it is not the one asked
|
||||
// for.
|
||||
func (b Build) versionOr(asked string) string {
|
||||
if b.Version != "" {
|
||||
return b.Version
|
||||
}
|
||||
return asked
|
||||
}
|
||||
|
||||
// as is a status for one build of what was asked for.
|
||||
func (st Status) as(w Want, b Build) Status {
|
||||
st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
|
||||
return st
|
||||
}
|
||||
|
||||
func (in *Installer) ready(st Status, b Build, bin, version string) Status {
|
||||
st.State, st.Binary = Ready, bin
|
||||
st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
|
||||
if b.Note != "" {
|
||||
st.Detail += "; " + b.Note
|
||||
}
|
||||
var missing []string
|
||||
for _, f := range b.CPUFlags {
|
||||
if !in.Host.HasCPUFlag(f) {
|
||||
@@ -299,9 +351,55 @@ func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, vers
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
|
||||
}
|
||||
if b.GPUBackend != "" {
|
||||
if err := in.verifyGPUBackend(tctx, dir, bin, b); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
}
|
||||
return bin, firstVersionLine(out), nil
|
||||
}
|
||||
|
||||
// verifyGPUBackend checks a GPU build can compute on this node's GPUs.
|
||||
//
|
||||
// The binary's own ldd says nothing about it: the GPU backend is a library
|
||||
// the server opens at run time, and when it cannot — cuBLAS missing, a driver
|
||||
// too old — the server carries on with its CPU backend. So the build is asked
|
||||
// which devices it sees, and when it sees none, ldd on the backend says why.
|
||||
func (in *Installer) verifyGPUBackend(ctx context.Context, dir, bin string, b Build) error {
|
||||
out, err := in.run(ctx, nil, bin, "--list-devices")
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s --list-devices failed: %v: %s", b.Binary, err, lastLine(out))
|
||||
}
|
||||
if listsDevice(string(out)) {
|
||||
return nil
|
||||
}
|
||||
why := "its GPU backend did not load"
|
||||
if lib, err := findFile(dir, b.GPUBackend); err != nil {
|
||||
why = "the build has no " + b.GPUBackend
|
||||
} else if out, err := in.run(ctx, nil, "ldd", lib); err == nil {
|
||||
if missing := lddMissing(string(out)); len(missing) > 0 {
|
||||
why = b.GPUBackend + " cannot load: " + strings.Join(missing, "; ")
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("the %s build of %s sees no GPU, and would run on the CPU: %s", b.Variant, b.Binary, why)
|
||||
}
|
||||
|
||||
// listsDevice reads llama-server --list-devices: a device line after
|
||||
// "Available devices:", which holds "(none)" when there is none.
|
||||
func listsDevice(out string) bool {
|
||||
listing := false
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
switch {
|
||||
case strings.HasPrefix(line, "Available devices"):
|
||||
listing = true
|
||||
case listing && line != "" && line != "(none)":
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// packages installs the operating system packages a build needs.
|
||||
//
|
||||
// Only those missing, and from the node's own package sources: a package the
|
||||
|
||||
@@ -89,7 +89,11 @@ type fakeSystem struct {
|
||||
installed map[string]bool
|
||||
tools map[string]bool
|
||||
lddOut string
|
||||
calls []string
|
||||
// devices is what llama-server --list-devices prints, and backendLdd
|
||||
// what ldd says of its GPU backend.
|
||||
devices string
|
||||
backendLdd string
|
||||
calls []string
|
||||
// locked is how many more package commands find the lock held.
|
||||
locked int
|
||||
}
|
||||
@@ -127,8 +131,12 @@ func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...str
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
case name == "ldd" && strings.HasSuffix(args[0], "libggml-cuda.so"):
|
||||
return []byte(f.backendLdd), nil
|
||||
case name == "ldd":
|
||||
return []byte(f.lddOut), nil
|
||||
case strings.HasSuffix(name, "llama-server") && args[0] == "--list-devices":
|
||||
return []byte(f.devices), nil
|
||||
case strings.HasSuffix(name, "llama-server"):
|
||||
return []byte("version: 10991 (abc)\n"), nil
|
||||
}
|
||||
@@ -247,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
|
||||
func TestTheBuildFollowsTheHardware(t *testing.T) {
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
b, err := chooseLlamaCpp("b1", gpu)
|
||||
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
|
||||
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
|
||||
got, err := llamaCppCandidates("b1", "", gpu)
|
||||
if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
|
||||
t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
|
||||
}
|
||||
if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
|
||||
}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
host discovery.Host
|
||||
want []string
|
||||
}{
|
||||
// Driver 580 and Turing: CUDA 13.3 first.
|
||||
{"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
|
||||
// A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
|
||||
{"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
|
||||
// An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
|
||||
{"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
|
||||
// A Pascal card: CUDA 12.8, not 13.3.
|
||||
{"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
|
||||
// arm64 with a GPU: never the CPU build.
|
||||
{"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
|
||||
} {
|
||||
got, err := llamaCppCandidates("b1", "", c.host)
|
||||
var names []string
|
||||
for _, b := range got {
|
||||
names = append(names, b.Variant)
|
||||
}
|
||||
if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
|
||||
t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
|
||||
}
|
||||
}
|
||||
|
||||
slow := ubuntu
|
||||
slow.CPUFlags = nil
|
||||
@@ -388,3 +421,32 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
|
||||
t.Errorf("link %q %v", target, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
|
||||
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
|
||||
// loader looks, and llama-server serving a GPU placement on the CPU.
|
||||
cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
|
||||
rel := newRelease(map[string][]byte{
|
||||
"llama-b1-" + cuda: tarGz(t, map[string]string{
|
||||
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
|
||||
"cudart-llama-b1-" + cuda: tarGz(t, map[string]string{"libcudart.so.12": "x"}, nil),
|
||||
})
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
|
||||
devices: "Available devices:\n (none)\n",
|
||||
backendLdd: "\tlibcublas.so.12 => not found\n\tlibcudart.so.12 => /x/libcudart.so.12\n"}
|
||||
in, _ := installer(t, rel.serve(t), sys, gpu)
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "sees no GPU") ||
|
||||
!strings.Contains(got[0].Detail, "libcublas.so.12 is not installed") {
|
||||
t.Fatalf("expected a failure naming the missing library: %+v", got[0])
|
||||
}
|
||||
|
||||
// The same build seeing its GPU is installed.
|
||||
sys.devices = "Available devices:\n CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}}); got[0].State != Ready {
|
||||
t.Fatalf("a CUDA build that sees its GPU: %+v", got[0])
|
||||
}
|
||||
}
|
||||
@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
|
||||
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
|
||||
defer cancel()
|
||||
// The version this build is, which for a GPU older than every build of
|
||||
// the one asked for is an older one: its publication date is the one
|
||||
// the dependencies are resolved as of.
|
||||
w.Version = b.versionOr(w.Version)
|
||||
|
||||
if err := in.packages(ctx, b.Packages, st); err != nil {
|
||||
return err
|
||||
@@ -86,28 +90,67 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
python := filepath.Join(tmp, "bin", "python")
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
|
||||
reqs := b.Requirements
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
// refuses a download that does not match it.
|
||||
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var asset, sum string
|
||||
for name, s := range sums {
|
||||
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
|
||||
asset, sum = name, s
|
||||
}
|
||||
}
|
||||
if asset == "" {
|
||||
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
|
||||
b.Wheel.Tag + "/" + asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
// is taken from whichever of the two has the version resolved — both are
|
||||
// the publishers' own — and each file is still pinned by its hash.
|
||||
var index []string
|
||||
if b.ExtraIndex != "" {
|
||||
index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
|
||||
}
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
|
||||
lock := filepath.Join(tmp, lockName)
|
||||
args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock, "-"}
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
|
||||
args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock}, index...)
|
||||
args = append(args, "-")
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
|
||||
strings.Join(b.Requirements, " "))
|
||||
if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
|
||||
strings.Join(reqs, " "))
|
||||
install := append([]string{"pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock}, index...)
|
||||
if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
in.progress(st, "checking that %s runs", w.Engine)
|
||||
if _, _, err := in.verify(ctx, tmp, b); err != nil {
|
||||
return err
|
||||
}
|
||||
if b.GPUCheck && in.Host.NVIDIADriver != "" {
|
||||
in.progress(st, "checking that %s (%s) computes on this node's GPU", w.Engine, b.Variant)
|
||||
if err := in.gpuCheck(ctx, python, w, b); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
m, _ := json.MarshalIndent(markerFile{
|
||||
Engine: w.Engine, Version: w.Version, Variant: b.Variant,
|
||||
Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
|
||||
Binary: filepath.Join("bin", b.Binary), Requirements: reqs,
|
||||
ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
|
||||
}, "", " ")
|
||||
if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
|
||||
@@ -203,3 +246,26 @@ func tail(b []byte) string {
|
||||
}
|
||||
return strings.Join(lines, " / ")
|
||||
}
|
||||
|
||||
// gpuProbe is what the environment is asked to run on the GPU: one small
|
||||
// operation, which needs the torch to have kernels for the card.
|
||||
const gpuProbe = `import torch
|
||||
assert torch.cuda.is_available(), "torch sees no CUDA device"
|
||||
x = torch.ones(64, device="cuda")
|
||||
print(float((x * 2).sum()), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))`
|
||||
|
||||
// gpuCheck runs gpuProbe in a freshly installed environment.
|
||||
func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Build) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
|
||||
defer cancel()
|
||||
out, err := in.run(ctx, nil, python, "-c", gpuProbe)
|
||||
if err != nil {
|
||||
hint := ""
|
||||
if strings.Contains(string(out), "no kernel image") {
|
||||
hint = "; this build has no kernels for this GPU, so try the other build for this node"
|
||||
}
|
||||
return fmt.Errorf("%s %s (%s) installed but cannot compute on this node's GPU: %s%s",
|
||||
w.Engine, w.Version, b.Variant, tail(out), hint)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package provision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
@@ -23,6 +24,10 @@ type fakePython struct {
|
||||
pmu sync.Mutex
|
||||
commands []string
|
||||
failPip bool
|
||||
// noKernel makes the GPU check fail as a torch without kernels for the
|
||||
// card does: in every build, or in noKernelIn only ("-cu13.").
|
||||
noKernel bool
|
||||
noKernelIn string
|
||||
}
|
||||
|
||||
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
@@ -43,6 +48,11 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
|
||||
if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
|
||||
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
|
||||
}
|
||||
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
|
||||
}
|
||||
return f.fakeSystem.run(ctx, env, name, args...)
|
||||
}
|
||||
@@ -59,6 +69,8 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
uv := newRelease(map[string][]byte{
|
||||
"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
|
||||
map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
|
||||
// The fake GitHub serves every repository's release from this one.
|
||||
"vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl": []byte("the cu129 wheel"),
|
||||
})
|
||||
gh := uv.serve(t)
|
||||
mux := http.NewServeMux()
|
||||
@@ -66,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
|
||||
{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
|
||||
})
|
||||
mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
|
||||
})
|
||||
pypi := httptest.NewServer(mux)
|
||||
t.Cleanup(pypi.Close)
|
||||
return &Installer{
|
||||
@@ -130,8 +145,10 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
says string
|
||||
}{
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
|
||||
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
// No vLLM was ever built for Pascal.
|
||||
{"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -182,3 +199,114 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
|
||||
t.Errorf("got %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
|
||||
// 12.9 build, from the project's GitHub release, with the torch PyTorch
|
||||
// publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, gh := pythonInstaller(t, volta, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("a Turing node on driver 550: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
|
||||
t.Errorf("not installed under its own build's name: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
wheel := "vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
|
||||
for _, want := range []string{
|
||||
// Pinned to the digest GitHub publishes for the wheel.
|
||||
"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
|
||||
strings.TrimPrefix(gh.digests[wheel], "sha256:"),
|
||||
"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
|
||||
"--require-hashes --no-deps -r",
|
||||
// Computing on the card, not only importing.
|
||||
"/bin/python -c import torch",
|
||||
} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("no %q in:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
|
||||
t.Errorf("installed without the index its torch comes from: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
|
||||
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
|
||||
}
|
||||
if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
|
||||
t.Error("a build that does not exist was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernel: true}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "no kernels for this GPU") {
|
||||
t.Fatalf("a build without kernels for the card: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu13", marker)); err == nil {
|
||||
t.Error("marked installed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
// A machine vLLM is not published for is refused, not given the other
|
||||
// build to fail with.
|
||||
if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
!strings.Contains(err.Error(), "no Linux build") {
|
||||
t.Errorf("an unknown architecture: %v", err)
|
||||
}
|
||||
if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
// Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
|
||||
got, err := vllmCandidates("0.30.0", "", gpuNode)
|
||||
if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
|
||||
t.Errorf("candidates on a current node: %+v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
|
||||
// 0.30.0 asked for; no build of it has Volta kernels. The node installs
|
||||
// 0.19.1, the newest that has, and says so.
|
||||
v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, v100, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
|
||||
!strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
|
||||
t.Fatalf("a V100: %+v", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
|
||||
t.Errorf("not installed as what it is: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
|
||||
t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
|
||||
}
|
||||
// Chosen cu13, which 0.19.1 has none of: refused, saying why.
|
||||
if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
|
||||
t.Error("a CUDA 13 build was offered to a V100")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
|
||||
// The table says CUDA 13 runs here; the card says otherwise. The next
|
||||
// build is installed rather than the node going without.
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Variant != "cu129" {
|
||||
t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -43,12 +43,41 @@ type Build struct {
|
||||
// VersionArgs is what Binary is run with to prove the build works, when
|
||||
// --version is not it.
|
||||
VersionArgs []string
|
||||
// GPUBackend, when set, is the shared library the build computes on GPUs
|
||||
// with, loaded at run time rather than linked: "libggml-cuda.so". The
|
||||
// build must list a device with it, or it is not installed — a llama.cpp
|
||||
// whose CUDA backend cannot load runs on the CPU without complaint.
|
||||
GPUBackend string
|
||||
// Wheel, when set, is the engine itself as a wheel from a GitHub
|
||||
// release rather than from PyPI, checked against the digest GitHub
|
||||
// publishes for it. ExtraIndex is a package index besides PyPI that the
|
||||
// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
|
||||
Wheel *Wheel
|
||||
ExtraIndex string
|
||||
// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
|
||||
// compute something on one before the build counts as installed. A build
|
||||
// with no kernels for the card — a GPU generation its torch dropped —
|
||||
// imports and prints its version, and fails on its first model.
|
||||
GPUCheck bool
|
||||
// Version is the engine version this build is, when it is not the one
|
||||
// asked for: the newest built for this machine's GPUs, where the one
|
||||
// asked for has no kernels for them. Empty is the version asked for.
|
||||
Version string
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
Project, Repo, Tag, Prefix, Suffix string
|
||||
}
|
||||
|
||||
// Recipe knows how to install one engine.
|
||||
type Recipe struct {
|
||||
// Repo is the GitHub repository whose releases carry the builds; empty
|
||||
@@ -56,8 +85,12 @@ type Recipe struct {
|
||||
Repo string
|
||||
// PyPI is the project an engine installed from PyPI is published as.
|
||||
PyPI string
|
||||
// Choose picks the build for a machine, or says why none fits.
|
||||
Choose func(version string, h discovery.Host) (Build, error)
|
||||
// Candidates are the builds that may run on a machine, best first, or
|
||||
// why none can. The installer takes the first that installs and is seen
|
||||
// to work — computing on the GPU, where there is one — so a build this
|
||||
// table is wrong about costs a download, not a node. variant is a build
|
||||
// chosen for the node; empty leaves the choice to the machine.
|
||||
Candidates func(version, variant string, h discovery.Host) ([]Build, error)
|
||||
}
|
||||
|
||||
// Recipes are the engines this agent can install.
|
||||
@@ -66,10 +99,10 @@ type Recipe struct {
|
||||
// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
|
||||
// checked against its published checksum.
|
||||
var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
|
||||
"vllm": {PyPI: "vllm", Choose: chooseVLLM},
|
||||
"sglang": {PyPI: "sglang", Choose: chooseSGLang},
|
||||
"uv": {Repo: "astral-sh/uv", Choose: chooseUV},
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
// Installable are the engines this agent can install, as it tells the
|
||||
@@ -99,7 +132,15 @@ type NotHere struct{ Reason string }
|
||||
|
||||
func (e NotHere) Error() string { return e.Reason }
|
||||
|
||||
func chooseUV(version string, h discovery.Host) (Build, error) {
|
||||
func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
|
||||
b, err := chooseUV(version, h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []Build{b}, nil
|
||||
}
|
||||
|
||||
func chooseUV(_ string, h discovery.Host) (Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -116,33 +157,227 @@ func chooseUV(version string, h discovery.Host) (Build, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
|
||||
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
|
||||
const minDriverForCUDA13 = 580
|
||||
// What builds need of the NVIDIA side of a machine, read from what each
|
||||
// project publishes and how it was built:
|
||||
//
|
||||
// - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
|
||||
// by CUDA's minor version compatibility — except code shipped only as
|
||||
// PTX, which the driver compiles and so needs a driver as new as the
|
||||
// CUDA it was compiled with.
|
||||
// - Which GPUs a build has kernels for is the build's own list, not what
|
||||
// its CUDA supports.
|
||||
const (
|
||||
minDriverForCUDA13 = 580
|
||||
minDriverForCUDA12 = 525
|
||||
)
|
||||
|
||||
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
// gpuNeeds is what one build needs of a machine's GPUs and driver.
|
||||
type gpuNeeds struct {
|
||||
driver int
|
||||
cc float64
|
||||
}
|
||||
|
||||
// cudaGate says whether a node can run an engine built on CUDA 13.
|
||||
func cudaGate(engine string, h discovery.Host) error {
|
||||
// fits says why a machine cannot run a build, or nil.
|
||||
func (n gpuNeeds) fits(what string, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
|
||||
return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
|
||||
return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// candidate is a build and what it needs.
|
||||
type candidate struct {
|
||||
b Build
|
||||
needs gpuNeeds
|
||||
what string
|
||||
}
|
||||
|
||||
// pick keeps the candidates a machine can run, in order, and those a
|
||||
// variant chosen for the node names; with none left, it says why each went.
|
||||
func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
|
||||
var out []Build
|
||||
var why []string
|
||||
for _, c := range cands {
|
||||
if variant != "" && c.b.Variant != variant {
|
||||
continue
|
||||
}
|
||||
if err := c.needs.fits(c.what, h); err != nil {
|
||||
why = append(why, err.Error())
|
||||
continue
|
||||
}
|
||||
out = append(out, c.b)
|
||||
}
|
||||
switch {
|
||||
case len(out) > 0:
|
||||
return out, nil
|
||||
case len(why) == 0 && variant != "":
|
||||
var names []string
|
||||
for _, c := range cands {
|
||||
names = append(names, c.b.Variant)
|
||||
}
|
||||
return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
|
||||
}
|
||||
return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
|
||||
}
|
||||
|
||||
// --- vLLM ---
|
||||
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for drivers CUDA 13 does not run on.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
|
||||
// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
|
||||
// "+cu129" beside a CUDA 13 one.
|
||||
const vllmCUDA13Since = "0.20"
|
||||
|
||||
// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
|
||||
// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
|
||||
// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
|
||||
// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
|
||||
// 0.20 and 8.0 since.
|
||||
func vllmMinCompute(version, arch string) float64 {
|
||||
old := versionBefore(version, vllmCUDA13Since)
|
||||
switch {
|
||||
case arch == "arm64" && old:
|
||||
return 8.7
|
||||
case arch == "arm64":
|
||||
return 8.0
|
||||
case old:
|
||||
return 7.0
|
||||
}
|
||||
return 7.5
|
||||
}
|
||||
|
||||
// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
|
||||
// on an architecture, for GPUs older than every build of the version asked
|
||||
// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
|
||||
// for a GPU no vLLM was built for.
|
||||
var vllmNewestFor = []struct {
|
||||
arch string
|
||||
from float64 // the oldest GPU it covers
|
||||
version string
|
||||
}{
|
||||
{"amd64", 7.0, "0.19.1"},
|
||||
}
|
||||
|
||||
// vllmCandidates are the vLLM builds for a machine.
|
||||
//
|
||||
// The version asked for, in the build the machine runs: CUDA 13 where the
|
||||
// driver allows, the same version's CUDA 12.9 build where it does not. One
|
||||
// version across the cluster wherever the GPUs allow it, because the flags
|
||||
// llmgw builds are checked against it. Where they do not — GPUs older than
|
||||
// every build of that version — the newest version built for them, said so
|
||||
// beside the installation: vLLM on a V100 at all, rather than not.
|
||||
func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
arch, err := gpuArch("vLLM", h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v, note := version, ""
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
|
||||
older := ""
|
||||
for _, n := range vllmNewestFor {
|
||||
if n.arch == h.Arch && cc >= n.from {
|
||||
older = n.version
|
||||
}
|
||||
}
|
||||
if older == "" {
|
||||
return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
|
||||
"the node serves llama.cpp models", cc, arch)
|
||||
}
|
||||
v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
|
||||
}
|
||||
pypi := func(variant string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + v},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
Note: note,
|
||||
}
|
||||
if v != version {
|
||||
b.Version = v
|
||||
}
|
||||
return b
|
||||
}
|
||||
minCC := vllmMinCompute(v, h.Arch)
|
||||
if versionBefore(v, vllmCUDA13Since) {
|
||||
// PyPI's own build is CUDA 12, and is the one there is; asked for
|
||||
// cu129, this is it.
|
||||
if variant == "cu129" {
|
||||
variant = "cu12"
|
||||
}
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
|
||||
}, h)
|
||||
}
|
||||
cu129 := pypi("cu129")
|
||||
cu129.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
cu129.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
|
||||
Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
|
||||
}
|
||||
cu129.ExtraIndex = pytorchCU129
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
|
||||
}, h)
|
||||
}
|
||||
|
||||
// gpuArch says whether a Python GPU engine is published for this machine
|
||||
// and whether it has an NVIDIA GPU at all.
|
||||
func gpuArch(engine string, h discovery.Host) (string, error) {
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
|
||||
return "", fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
|
||||
}
|
||||
if h.NVIDIADriver == "" {
|
||||
return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
|
||||
return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, minDriverForCUDA13, h.NVIDIADriver)
|
||||
return h.Arch, nil
|
||||
}
|
||||
|
||||
// --- SGLang ---
|
||||
|
||||
// sglangCUDA13Since is the first SGLang built for CUDA 13.
|
||||
const sglangCUDA13Since = "0.5"
|
||||
|
||||
func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("SGLang", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
|
||||
return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
|
||||
engine, minComputeForCUDA13, cc)
|
||||
b := Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
return nil
|
||||
needs := gpuNeeds{minDriverForCUDA13, 7.5}
|
||||
if versionBefore(version, sglangCUDA13Since) {
|
||||
b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
|
||||
}
|
||||
return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
@@ -150,35 +385,27 @@ func cudaGate(engine string, h discovery.Host) error {
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
func chooseVLLM(version string, h discovery.Host) (Build, error) {
|
||||
if err := cudaGate("vLLM", h); err != nil {
|
||||
return Build{}, err
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
func versionBefore(a, b string) bool {
|
||||
if b == "" {
|
||||
return false
|
||||
}
|
||||
return Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func chooseSGLang(version string, h discovery.Host) (Build, error) {
|
||||
if err := cudaGate("SGLang", h); err != nil {
|
||||
return Build{}, err
|
||||
pa, pb := strings.Split(strings.TrimPrefix(a, "v"), "."), strings.Split(b, ".")
|
||||
for i := 0; i < max(len(pa), len(pb)); i++ {
|
||||
x, y := 0, 0
|
||||
if i < len(pa) {
|
||||
x = leadingInt(pa[i])
|
||||
}
|
||||
if i < len(pb) {
|
||||
y = leadingInt(pb[i])
|
||||
}
|
||||
if x != y {
|
||||
return x < y
|
||||
}
|
||||
}
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
return Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
return false
|
||||
}
|
||||
|
||||
// leadingInt is the number a version string starts with: 580 of "580.65.06".
|
||||
@@ -193,17 +420,30 @@ func leadingInt(s string) int {
|
||||
return n
|
||||
}
|
||||
|
||||
// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
|
||||
// carries a matching cudart archive, so the node needs a driver and nothing
|
||||
// else — no toolkit.
|
||||
const cudaForLlamaCpp = "12.8"
|
||||
// --- llama.cpp ---
|
||||
|
||||
func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them
|
||||
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
|
||||
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
|
||||
// name decides nothing here. The build is tried everywhere, and ldd says
|
||||
// whether this system can load it — in the words of the missing symbol.
|
||||
// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
|
||||
// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
|
||||
// carries as PTX, which a driver older than the CUDA it came from cannot
|
||||
// compile.
|
||||
var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
|
||||
|
||||
// llamaCppCandidates are the llama.cpp builds for a machine.
|
||||
//
|
||||
// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
|
||||
// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
|
||||
// the GPUs it ships as PTX, 525 for those it ships as machine code), and
|
||||
// Vulkan, which runs on any card whose driver has Vulkan — the generations
|
||||
// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
|
||||
// a GPU placement served on the CPU, which is what installing checks for.
|
||||
// The CPU build where there is no NVIDIA GPU.
|
||||
//
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them is
|
||||
// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
|
||||
// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
|
||||
// nothing here. The build is tried everywhere, and ldd says whether this
|
||||
// system can load it — in the words of the missing symbol.
|
||||
func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -211,35 +451,53 @@ func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
|
||||
case "arm64":
|
||||
arch = "arm64"
|
||||
default:
|
||||
return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
|
||||
b := Build{
|
||||
Variant: "cpu",
|
||||
Assets: []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
|
||||
// Everything else the build links is in the archive or in any current
|
||||
// glibc system; OpenMP's runtime is the one a cloud image lacks, and
|
||||
// the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the best
|
||||
// one, down to SSE4.2. Below AVX2 it runs, several times slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
// The CUDA build is published for x64 at this version; on arm64 it is a
|
||||
// newer CUDA that the driver may not support, so an arm64 GPU node runs
|
||||
// the CPU build until someone decides otherwise.
|
||||
if h.NVIDIADriver != "" && arch == "x64" {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
|
||||
b.Variant = "cuda-" + cudaForLlamaCpp
|
||||
b.Assets = []string{
|
||||
fmt.Sprintf("llama-%s-%s", version, name),
|
||||
fmt.Sprintf("cudart-llama-%s-%s", version, name),
|
||||
base := func(variant string, assets ...string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Assets: assets,
|
||||
// Everything else the build links is in the archive or in any
|
||||
// current glibc system; OpenMP's runtime is the one a cloud image
|
||||
// lacks, and the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the
|
||||
// best one, down to SSE4.2. Below AVX2 it runs, several times
|
||||
// slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
return b
|
||||
}
|
||||
return b, nil
|
||||
if h.NVIDIADriver == "" {
|
||||
return pick("llama.cpp", variant, []candidate{
|
||||
{base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
|
||||
}, h)
|
||||
}
|
||||
cuda := func(cv string) Build {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
|
||||
// The release carries the CUDA runtime beside the build, so the node
|
||||
// needs a driver and nothing else — no toolkit.
|
||||
b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
return b
|
||||
}
|
||||
vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
|
||||
vulkan.GPUBackend = "libggml-vulkan.so"
|
||||
vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
|
||||
|
||||
cuda12Driver := 570
|
||||
if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
|
||||
cuda12Driver = minDriverForCUDA12
|
||||
}
|
||||
cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
|
||||
if arch == "x64" {
|
||||
cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
|
||||
}
|
||||
cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
|
||||
return pick("llama.cpp", variant, cands, h)
|
||||
}
|
||||
|
||||
func orUnknown(s string) string {
|
||||
|
||||
@@ -2,6 +2,8 @@ package weights
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -26,8 +28,9 @@ type manifest struct {
|
||||
Model string `json:"model"`
|
||||
Version string `json:"version"`
|
||||
Files []struct {
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
} `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
@@ -87,6 +90,25 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error)
|
||||
}
|
||||
}
|
||||
|
||||
// Each file against the checksum the mirror took when it fetched it,
|
||||
// before the set is called complete: a shard damaged on the way loads as
|
||||
// a model that answers nonsense, not as an error.
|
||||
for _, file := range m.Files {
|
||||
if file.SHA256 == "" {
|
||||
continue
|
||||
}
|
||||
target := filepath.Join(dir, file.Name)
|
||||
got, err := fileSHA256(target)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !strings.EqualFold(got, file.SHA256) {
|
||||
_ = os.Remove(target)
|
||||
return "", fmt.Errorf("%s%s: sha256 %s, the mirror has %s; removed, to be fetched again",
|
||||
source, file.Name, got, file.SHA256)
|
||||
}
|
||||
}
|
||||
|
||||
if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -108,6 +130,12 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
|
||||
return manifest{}, fmt.Errorf("asking %s what it holds: %w", source, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode == http.StatusConflict {
|
||||
// Still downloading on the mirror, or a shard missing: said in the
|
||||
// mirror's words, which name what is not there yet.
|
||||
why, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
|
||||
return manifest{}, fmt.Errorf("%s is not complete on the mirror yet: %s", source, strings.TrimSpace(string(why)))
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return manifest{}, fmt.Errorf("%s answered %d", source, resp.StatusCode)
|
||||
}
|
||||
@@ -127,3 +155,16 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func fileSHA256(path string) (string, error) {
|
||||
f, err := os.Open(path) //nolint:gosec // a file this fetcher wrote
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
if _, err := io.Copy(h, f); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), nil
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
package weights
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -165,3 +167,40 @@ func TestOnlyATrailingSlashMeansASet(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetFileThatDoesNotMatchTheMirrorsChecksumIsRefused(t *testing.T) {
|
||||
good := "the shard as mirrored"
|
||||
sum := sha256.Sum256([]byte(good))
|
||||
served := "the shard damaged on the way"
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/") {
|
||||
fmt.Fprintf(w, `{"model":"m","version":"v","files":[{"name":"s.safetensors","bytes":%d,"sha256":%q}]}`,
|
||||
len(served), hex.EncodeToString(sum[:]))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(served))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
f := &Fetcher{CacheDir: t.TempDir()}
|
||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err == nil || !strings.Contains(err.Error(), "sha256") {
|
||||
t.Fatalf("a damaged shard was accepted: %v", err)
|
||||
}
|
||||
served = good
|
||||
// The size differs, so it is fetched again; this time it matches.
|
||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
|
||||
t.Fatalf("the shard as mirrored: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetStillDownloadingOnTheMirrorSaysSo(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.Error(w, "m/v: this version is not complete: still downloading model-00002-of-00002.safetensors", http.StatusConflict)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
f := &Fetcher{CacheDir: t.TempDir()}
|
||||
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
|
||||
if err == nil || !strings.Contains(err.Error(), "not complete on the mirror yet") ||
|
||||
!strings.Contains(err.Error(), "model-00002-of-00002") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
}
|
||||
@@ -143,6 +143,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
newEngineCheckCmd(),
|
||||
newEngineBuildCmd(client),
|
||||
newToolCheckCmd(),
|
||||
newWeightsCmd(client, &asJSON),
|
||||
newCapacityCmd(client, &asJSON),
|
||||
|
||||
@@ -12,8 +12,9 @@ import (
|
||||
|
||||
func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
var (
|
||||
models []string
|
||||
all bool
|
||||
models []string
|
||||
all bool
|
||||
inPlace bool
|
||||
)
|
||||
c := &cobra.Command{
|
||||
Use: "restart [REPLICA...]",
|
||||
@@ -22,10 +23,14 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
"an edited profile, a new engine version.\n\n" +
|
||||
"Each replacement is placed beside the replica it replaces, and the old one\n" +
|
||||
"keeps serving until the new one is ready. Replicas of one model are\n" +
|
||||
"replaced one at a time. Where nothing else has room for a second copy —\n" +
|
||||
"the usual case on a small CPU fleet — the old one is stopped after the\n" +
|
||||
"drain grace period (5 minutes by default) and the new one starts in its\n" +
|
||||
"place; a model with only that replica is unserved while it loads.\n\n" +
|
||||
"replaced one at a time.\n\n" +
|
||||
"Where there is no room for a second copy and the old replica is all that\n" +
|
||||
"is in the way — one GPU, every card that fits held — it is stopped at once\n" +
|
||||
"and the new one starts in its place; a model with only that replica is\n" +
|
||||
"unserved while it loads. Where the new one would not fit even then — a\n" +
|
||||
"profile edited to need more than any node has — the restart is given up\n" +
|
||||
"and the old replica keeps serving: llmgw ctl replicas says so, and why.\n\n" +
|
||||
"--in-place stops each replica first, whatever room there is.\n\n" +
|
||||
" llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
|
||||
" llmgw ctl restart --model qwen2.5-7b\n" +
|
||||
" llmgw ctl restart --all",
|
||||
@@ -33,7 +38,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
if len(args) == 0 && len(models) == 0 && !all {
|
||||
return errors.New("name replicas, --model, or --all")
|
||||
}
|
||||
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
|
||||
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all, InPlace: inPlace})
|
||||
var res adminapi.RestartResult
|
||||
if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
|
||||
return err
|
||||
@@ -44,6 +49,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
}
|
||||
c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
|
||||
c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
|
||||
c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before starting its replacement")
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
+17
-2
@@ -23,6 +23,7 @@ func newDoctorCmd() *cobra.Command {
|
||||
opts doctor.Options
|
||||
asJSON bool
|
||||
verbose bool
|
||||
fabrics []string
|
||||
)
|
||||
cmd := &cobra.Command{
|
||||
Use: "doctor",
|
||||
@@ -38,8 +39,21 @@ that file when the report is not enough to go on.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
doctor.SetConfigLoader(config.LoadAgent)
|
||||
if opts.VLLMVersion == "" {
|
||||
for _, spec := range fabrics {
|
||||
name, hcas, _ := strings.Cut(spec, ":")
|
||||
if name == "" || hcas == "" {
|
||||
return fmt.Errorf("--fabric %q: give it as name:mlx5_0,mlx5_1", spec)
|
||||
}
|
||||
if opts.Fabrics == nil {
|
||||
opts.Fabrics = map[string][]string{}
|
||||
}
|
||||
opts.Fabrics[name] = strings.Split(hcas, ",")
|
||||
}
|
||||
switch opts.VLLMVersion {
|
||||
case "":
|
||||
opts.VLLMVersion = config.DefaultEngines["vllm"]
|
||||
case "none":
|
||||
opts.VLLMVersion = ""
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
@@ -66,7 +80,8 @@ that file when the report is not enough to go on.`,
|
||||
f.StringVar(&opts.ConfigPath, "config", "", "agent configuration (default /etc/llmgw/agent.yaml when present)")
|
||||
f.StringVar(&opts.IBRoot, "ib-root", discovery.SysfsRoot, "sysfs InfiniBand directory")
|
||||
f.StringVar(&opts.NvidiaSMI, "nvidia-smi", "", "path to nvidia-smi")
|
||||
f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs)")
|
||||
f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs; none to skip)")
|
||||
f.StringArrayVar(&fabrics, "fabric", nil, "a fabric about to be declared, name:mlx5_0,mlx5_1, checked as if it were; repeatable")
|
||||
f.BoolVar(&asJSON, "json", false, "write every fact and finding as JSON, to send back")
|
||||
f.BoolVar(&verbose, "verbose", false, "also print the raw nvidia-smi topology")
|
||||
return cmd
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "engine-build <node> <engine> <build>",
|
||||
Short: "Choose which build of an engine a GPU node installs",
|
||||
Long: "A node installs the build of vLLM its hardware runs: CUDA 13 where the\n" +
|
||||
"driver and GPUs allow, else the same version built for CUDA 12.9. This\n" +
|
||||
"chooses instead, as the cluster page's vLLM build column does:\n\n" +
|
||||
" auto what the hardware runs (the default)\n" +
|
||||
" cu13 the CUDA 13 build, from PyPI\n" +
|
||||
" cu129 the CUDA 12.9 build, from vLLM's GitHub release\n" +
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
|
||||
"gets 0.19.1, the newest built for it, by itself.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
body, _ := json.Marshal(adminapi.NodeEngineRequest{Engine: args[1], Build: args[2]})
|
||||
if err := client().post(cmd.Context(), "/api/v1/nodes/"+args[0]+"/engines", body, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s build %s; the node installs it in the background (llmgw ctl nodes shows it)\n",
|
||||
args[0], args[1], args[2])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
@@ -126,6 +127,14 @@ func newEngineCheckCmd() *cobra.Command {
|
||||
}
|
||||
sort.Strings(unknown)
|
||||
|
||||
// Flags accepted is not the same as running where it is placed: a
|
||||
// GPU build whose backend does not load accepts every flag and
|
||||
// computes on the CPU. Asked of the engine here, not of a help.
|
||||
var deviceErr error
|
||||
if fb, ok := eng.(engines.CPUFallback); ok && help == "" {
|
||||
deviceErr = checkDevices(cmd.Context(), name, fb, out)
|
||||
}
|
||||
|
||||
fmt.Fprintf(out, "%s: %d flags checked", name, checked)
|
||||
if skipped > 0 {
|
||||
fmt.Fprintf(out, ", %d commands skipped", skipped)
|
||||
@@ -137,7 +146,7 @@ func newEngineCheckCmd() *cobra.Command {
|
||||
return fmt.Errorf("nothing was checked; install the engine here or pass --help-output")
|
||||
}
|
||||
fmt.Fprintln(out, "every flag llmgw builds is accepted")
|
||||
return nil
|
||||
return deviceErr
|
||||
}
|
||||
fmt.Fprintf(out, "\n%d flags not mentioned by the program they are passed to:\n", len(unknown))
|
||||
for _, f := range unknown {
|
||||
@@ -249,3 +258,22 @@ func isFlagChar(b byte) bool {
|
||||
return b == '-' || b == '_' || (b >= 'a' && b <= 'z') ||
|
||||
(b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9')
|
||||
}
|
||||
|
||||
// checkDevices says which GPUs an engine that can fall back to the CPU sees on
|
||||
// this machine, and fails when the machine has NVIDIA GPUs and it sees none.
|
||||
func checkDevices(ctx context.Context, name string, fb engines.CPUFallback, out io.Writer) error {
|
||||
if _, err := exec.LookPath("nvidia-smi"); err != nil {
|
||||
fmt.Fprintln(out, "no NVIDIA GPU here: which devices it computes on was not checked")
|
||||
return nil
|
||||
}
|
||||
devs, err := fb.Devices(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(devs) == 0 {
|
||||
return fmt.Errorf("this machine has NVIDIA GPUs and %s sees none: it would serve a GPU placement on the CPU. "+
|
||||
"Its GPU backend did not load; ldd on its libggml-cuda.so names the missing library", name)
|
||||
}
|
||||
fmt.Fprintf(out, "computes on: %s\n", strings.Join(devs, "; "))
|
||||
return nil
|
||||
}
|
||||
+54
-29
@@ -2,6 +2,7 @@ package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
@@ -56,52 +57,69 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
|
||||
var model, version, as string
|
||||
|
||||
c := &cobra.Command{
|
||||
Use: "add <url>",
|
||||
Use: "add <url>...",
|
||||
Short: "Download a model into the mirror",
|
||||
Long: "Downloads to a temporary name and renames when complete, so an\n" +
|
||||
"interrupted download never leaves something a catalog could be pointed\n" +
|
||||
"at. Reports the checksum: a mirror is the one place in this system where\n" +
|
||||
"a corrupted file is copied to every node rather than caught once.\n\n" +
|
||||
"Several URLs go into one version, as the files of one model: the shards\n" +
|
||||
"of a safetensors set with its config.json and tokenizer, for vLLM. The\n" +
|
||||
"version is offered to nodes only once it is whole — nothing still\n" +
|
||||
"downloading, every shard its index names present — and its source is\n" +
|
||||
"the version, ending in a slash.\n\n" +
|
||||
"Refuses to replace a version that is already here. Add a new one beside\n" +
|
||||
"it instead — that is what makes the old one still available when the new\n" +
|
||||
"one turns out to be worse.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
source := args[0]
|
||||
file := as
|
||||
if file == "" {
|
||||
file = mirror.LastPathElement(source)
|
||||
}
|
||||
if model == "" || version == "" {
|
||||
return fmt.Errorf("--model and --version are required: they are the layout")
|
||||
}
|
||||
|
||||
e := mirror.Entry{Model: model, Version: version, File: file}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "fetching %s\n", source)
|
||||
|
||||
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
|
||||
if total > 0 {
|
||||
fmt.Fprintf(out, " %s of %s (%d%%)\n",
|
||||
fileSize(done), fileSize(total), done*100/total)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, " %s so far\n", fileSize(done))
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
if as != "" && len(args) > 1 {
|
||||
return fmt.Errorf("--as names one file; with %d URLs each keeps its own name", len(args))
|
||||
}
|
||||
fmt.Fprintf(out, "\n%s/%s/%s %s\n sha256 %s\n",
|
||||
added.Entry.Model, added.Entry.Version, added.Entry.File,
|
||||
fileSize(added.Entry.Bytes), added.SHA256)
|
||||
fmt.Fprintf(out, "\nPoint a model at it with:\n source: %s\n",
|
||||
added.Entry.URL("http://<this-host>:8081"))
|
||||
out := cmd.OutOrStdout()
|
||||
var last mirror.Added
|
||||
for _, source := range args {
|
||||
file := as
|
||||
if file == "" {
|
||||
file = mirror.LastPathElement(source)
|
||||
}
|
||||
e := mirror.Entry{Model: model, Version: version, File: file}
|
||||
fmt.Fprintf(out, "fetching %s\n", source)
|
||||
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
|
||||
if total > 0 {
|
||||
fmt.Fprintf(out, " %s of %s (%d%%)\n",
|
||||
fileSize(done), fileSize(total), done*100/total)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, " %s so far\n", fileSize(done))
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s/%s %s\n sha256 %s\n",
|
||||
added.Entry.Model, added.Entry.Version, added.Entry.File,
|
||||
fileSize(added.Entry.Bytes), added.SHA256)
|
||||
last = added
|
||||
}
|
||||
src := last.Entry.URL("http://<this-host>:8081")
|
||||
if len(args) > 1 {
|
||||
src = strings.TrimSuffix(mirror.Entry{Model: model, Version: version}.URL("http://<this-host>:8081"), "/") + "/"
|
||||
if m, err := mirror.ManifestFor(*dir, model, version); err != nil {
|
||||
fmt.Fprintf(out, "\nNot yet offered to nodes: %v\n", err)
|
||||
} else {
|
||||
fmt.Fprintf(out, "\n%d files, %s, whole.\n", len(m.Files), fileSize(m.Bytes))
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(out, "\nPoint a model at it with:\n source: %s\n", src)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
|
||||
c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
|
||||
c.Flags().StringVar(&as, "as", "", "the filename to save as; taken from the URL when omitted")
|
||||
c.Flags().StringVar(&as, "as", "", "the filename to save as, for one URL; taken from the URL when omitted")
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -405,7 +423,14 @@ func withManifest(root string, next http.Handler) http.Handler {
|
||||
return
|
||||
}
|
||||
m, err := mirror.ManifestFor(root, model, version)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, mirror.ErrIncomplete):
|
||||
// Said, not hidden behind a 404: a node asking now is told why
|
||||
// it cannot have the version yet, and tries again later.
|
||||
w.Header().Set("Retry-After", "60")
|
||||
http.Error(w, err.Error(), http.StatusConflict)
|
||||
return
|
||||
case err != nil:
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
+20
-1
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
@@ -113,13 +114,14 @@ engines are kept.`,
|
||||
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
|
||||
f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)")
|
||||
f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs")
|
||||
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml")
|
||||
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed, instead of the one the agent installs")
|
||||
f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed")
|
||||
f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH")
|
||||
f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband")
|
||||
f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority")
|
||||
f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
|
||||
f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
|
||||
f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
|
||||
f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
|
||||
f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
|
||||
f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp")
|
||||
@@ -148,6 +150,23 @@ func (na *nodeAdd) prepare() error {
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, e := range o.DisableEngines {
|
||||
if !slices.Contains(provision.Installable(), e) {
|
||||
return fmt.Errorf("--disable-engine %s: the engines are %s", e, strings.Join(provision.Installable(), ", "))
|
||||
}
|
||||
}
|
||||
// What the control plane will ask the node to install, so doctor checks
|
||||
// that vLLM rather than its own default. Its overrides for a GPU class
|
||||
// are not known until the node says what it has, and are not applied
|
||||
// here; doctor's verdict on vLLM is a warning either way.
|
||||
if o.VLLMVersion == "" {
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil {
|
||||
o.VLLMVersion = cfg.WantedEngines()["vllm"]
|
||||
if o.VLLMVersion == "" {
|
||||
o.VLLMVersion = "none"
|
||||
}
|
||||
}
|
||||
}
|
||||
o.Version = version.Version
|
||||
if o.Binary == "" {
|
||||
exe, err := os.Executable()
|
||||
|
||||
+18
-6
@@ -5,19 +5,23 @@ import (
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net"
|
||||
"net/http"
|
||||
"os/signal"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"golang.org/x/net/http2"
|
||||
"golang.org/x/net/http2/h2c"
|
||||
@@ -491,13 +495,21 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
// And what each model was measured to serve, for scaling and for
|
||||
// choosing between profiles.
|
||||
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
|
||||
// The engine builds chosen per node on the cluster page.
|
||||
ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
}
|
||||
svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
|
||||
svc.SetEngines(cfg.WantedEngines())
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines())
|
||||
svc.SetEnginesFor(func(n inventory.Node) map[string]string {
|
||||
return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
|
||||
})
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
|
||||
if ch := ctl.EngineChoices(); ch != nil {
|
||||
svc.SetEngineChoices(ch.For)
|
||||
ch.OnChange(svc.Reprovision)
|
||||
}
|
||||
// A standby must refuse agents as well as refusing to place. Commands
|
||||
// travel down the stream the agent opened, so a standby holding one is a
|
||||
// node the leader cannot command — and the leader's placements would fail
|
||||
|
||||
@@ -8,6 +8,7 @@ package config
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -85,6 +86,13 @@ type Server struct {
|
||||
// one leaves that engine to be installed by hand.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
|
||||
// EngineOverrides give some nodes different engines from the rest: a
|
||||
// vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
// all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
// engines over the cluster's, in order, later ones last; an empty version
|
||||
// removes that engine from the nodes it picks.
|
||||
EngineOverrides []EngineOverride `koanf:"engine_overrides"`
|
||||
|
||||
// BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
// Empty means /var/backups/llmgw-server.
|
||||
@@ -548,6 +556,14 @@ func Load(path string) (Server, error) {
|
||||
// Validate rejects configurations that would otherwise fail confusingly at
|
||||
// request time.
|
||||
func (s Server) Validate() error {
|
||||
for i, o := range s.EngineOverrides {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)
|
||||
}
|
||||
if len(o.Engines) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] changes no engine: give it engines, with \"\" for none", i)
|
||||
}
|
||||
}
|
||||
if s.Listen == "" {
|
||||
return fmt.Errorf("listen is empty")
|
||||
}
|
||||
@@ -626,6 +642,58 @@ var DefaultEngines = map[string]string{
|
||||
"vllm": "0.30.0",
|
||||
}
|
||||
|
||||
// EngineOverride gives the nodes it picks different engines. Every selector
|
||||
// set must match; at least one must be set.
|
||||
type EngineOverride struct {
|
||||
// GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
GPUClass string `koanf:"gpu_class"`
|
||||
// Nodes picks nodes by id.
|
||||
Nodes []string `koanf:"nodes"`
|
||||
// Labels picks nodes carrying all of these.
|
||||
Labels map[string]string `koanf:"labels"`
|
||||
// Engines are laid over the cluster's: a version, or "" for none.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
}
|
||||
|
||||
// Picks reports whether the override applies to a node with this id, these
|
||||
// GPU classes and these labels.
|
||||
func (o EngineOverride) Picks(node string, classes []string, labels map[string]string) bool {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return false
|
||||
}
|
||||
if o.GPUClass != "" && !slices.Contains(classes, o.GPUClass) {
|
||||
return false
|
||||
}
|
||||
if len(o.Nodes) > 0 && !slices.Contains(o.Nodes, node) {
|
||||
return false
|
||||
}
|
||||
for k, v := range o.Labels {
|
||||
if labels[k] != v {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// EnginesFor is what one node installs: WantedEngines with every override
|
||||
// that picks it laid over, in order.
|
||||
func (s Server) EnginesFor(node string, classes []string, labels map[string]string) map[string]string {
|
||||
out := s.WantedEngines()
|
||||
for _, o := range s.EngineOverrides {
|
||||
if !o.Picks(node, classes, labels) {
|
||||
continue
|
||||
}
|
||||
for k, v := range o.Engines {
|
||||
if v == "" {
|
||||
delete(out, k)
|
||||
continue
|
||||
}
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// WantedEngines is DefaultEngines with the configuration laid over it.
|
||||
func (s Server) WantedEngines() map[string]string {
|
||||
out := map[string]string{}
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEachNodeGetsTheEnginesItsOverridesSay(t *testing.T) {
|
||||
s := Server{
|
||||
Engines: map[string]string{"vllm": "0.30.0"},
|
||||
EngineOverrides: []EngineOverride{
|
||||
// Volta: CUDA 13 dropped it, CUDA 12 vLLM still runs.
|
||||
{GPUClass: "v100-32g", Engines: map[string]string{"vllm": "0.25.1"}},
|
||||
// Nodes meant for llama.cpp alone.
|
||||
{Labels: map[string]string{"role": "gguf"}, Engines: map[string]string{"vllm": ""}},
|
||||
},
|
||||
}
|
||||
if got := s.EnginesFor("h1", []string{"h100-80g"}, nil); got["vllm"] != "0.30.0" || got["llamacpp"] == "" {
|
||||
t.Errorf("an H100 node: %v", got)
|
||||
}
|
||||
if got := s.EnginesFor("v1", []string{"v100-32g"}, nil); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("a V100 node: %v", got)
|
||||
}
|
||||
got := s.EnginesFor("v2", []string{"v100-32g"}, map[string]string{"role": "gguf"})
|
||||
if _, has := got["vllm"]; has || got["llamacpp"] == "" {
|
||||
t.Errorf("a V100 node for llama.cpp alone, the later override last: %v", got)
|
||||
}
|
||||
|
||||
for _, bad := range []EngineOverride{
|
||||
{Engines: map[string]string{"vllm": ""}},
|
||||
{GPUClass: "v100-32g"},
|
||||
} {
|
||||
s := Server{Listen: ":8080", EngineOverrides: []EngineOverride{bad}}
|
||||
if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "engine_overrides[0]") {
|
||||
t.Errorf("%+v: %v", bad, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ package adminapi
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
@@ -111,6 +112,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/engines", a.setNodeEngine)
|
||||
mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
|
||||
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
|
||||
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
|
||||
@@ -156,6 +158,10 @@ type NodeView struct {
|
||||
// Provision is where installing each wanted engine stands. An engine is in
|
||||
// Engines only once its installation here is ready.
|
||||
Provision []inventory.EngineInstall `json:"provision,omitempty"`
|
||||
// EngineBuilds are the builds chosen for this node, by engine: "auto"
|
||||
// (its hardware decides), a build ("cu129"), or "none". Only on a node
|
||||
// with GPUs, the only kind with a choice to make.
|
||||
EngineBuilds map[string]string `json:"engine_builds,omitempty"`
|
||||
// OS is the operating system and architecture, as the agent read them.
|
||||
OS string `json:"os,omitempty"`
|
||||
// Warnings are facts about the machine that make it slow or unusable
|
||||
@@ -208,10 +214,13 @@ type NodeView struct {
|
||||
|
||||
// GPUView is one card as a node's page shows it.
|
||||
type GPUView struct {
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
NUMA int `json:"numa"`
|
||||
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
// ComputeCapability is what decides which engine builds run on the card:
|
||||
// CUDA 13 builds need 7.5, so a V100 (7.0) runs only older ones.
|
||||
ComputeCapability string `json:"compute_capability,omitempty"`
|
||||
NUMA int `json:"numa"`
|
||||
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
|
||||
// NearestNIC is the adapter closest to the card, and how close.
|
||||
NearestNIC string `json:"nearest_nic,omitempty"`
|
||||
NICPath string `json:"nic_path,omitempty"`
|
||||
@@ -470,6 +479,13 @@ func (a *API) Nodes() []NodeView {
|
||||
}
|
||||
slices.Sort(v.Engines)
|
||||
v.Provision = n.Provision
|
||||
if ch := a.ctl.EngineChoices(); ch != nil && len(n.GPUs) > 0 {
|
||||
chosen := ch.For(n.ID)
|
||||
v.EngineBuilds = map[string]string{"vllm": "auto"}
|
||||
for e, b := range chosen {
|
||||
v.EngineBuilds[e] = b
|
||||
}
|
||||
}
|
||||
if h := n.Host; h.OSID != "" {
|
||||
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
|
||||
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
|
||||
@@ -497,7 +513,8 @@ func (a *API) Nodes() []NodeView {
|
||||
v.Load = loadView(n.Load, time.Now())
|
||||
v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
|
||||
for _, g := range n.GPUs {
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, ComputeCapability: g.ComputeCapability,
|
||||
NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
for hca, rel := range g.NICPaths {
|
||||
if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
|
||||
(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
|
||||
@@ -1238,6 +1255,9 @@ type RestartRequest struct {
|
||||
Replicas []string `json:"replicas,omitempty"`
|
||||
Models []string `json:"models,omitempty"`
|
||||
All bool `json:"all,omitempty"`
|
||||
// InPlace stops each replica before starting its replacement, rather
|
||||
// than starting the replacement beside it first.
|
||||
InPlace bool `json:"in_place,omitempty"`
|
||||
}
|
||||
|
||||
// RestartResult lists the replicas queued.
|
||||
@@ -1261,7 +1281,11 @@ func (a *API) restart(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
|
||||
return
|
||||
}
|
||||
if err := a.ctl.Reconciler().Restart(ids...); err != nil {
|
||||
restart := a.ctl.Reconciler().Restart
|
||||
if req.InPlace {
|
||||
restart = a.ctl.Reconciler().RestartInPlace
|
||||
}
|
||||
if err := restart(ids...); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
@@ -1521,3 +1545,47 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
|
||||
}
|
||||
|
||||
// NodeEngineRequest chooses one engine's build for a node.
|
||||
type NodeEngineRequest struct {
|
||||
Engine string `json:"engine"`
|
||||
// Build is "auto", a build such as "cu129", or "none".
|
||||
Build string `json:"build"`
|
||||
}
|
||||
|
||||
// setNodeEngine chooses an engine's build for a node, and has the node
|
||||
// install it.
|
||||
func (a *API) setNodeEngine(w http.ResponseWriter, r *http.Request) {
|
||||
ch := a.ctl.EngineChoices()
|
||||
if ch == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the choice in")
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if _, ok := a.ctl.Inventory().Get(id); !ok {
|
||||
writeError(w, http.StatusNotFound, "no node "+id)
|
||||
return
|
||||
}
|
||||
var req NodeEngineRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := ch.Set(r.Context(), id, req.Engine, req.Build, actorOf(r)); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, req)
|
||||
}
|
||||
|
||||
// SetNodeEngine chooses an engine's build for a node, for the cluster page.
|
||||
func (a *API) SetNodeEngine(ctx context.Context, node, engine, build, by string) error {
|
||||
ch := a.ctl.EngineChoices()
|
||||
if ch == nil {
|
||||
return errors.New("this control plane has no store to keep the choice in")
|
||||
}
|
||||
if _, ok := a.ctl.Inventory().Get(node); !ok {
|
||||
return errors.New("no node " + node)
|
||||
}
|
||||
return ch.Set(ctx, node, engine, build, by)
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// choiceStore keeps engine choices in memory.
|
||||
type choiceStore struct{ m map[string]map[string]string }
|
||||
|
||||
func (c *choiceStore) NodeEngines(context.Context) (map[string]map[string]string, error) {
|
||||
return c.m, nil
|
||||
}
|
||||
func (c *choiceStore) SetNodeEngine(_ context.Context, node, engine, variant, _ string) error {
|
||||
if c.m[node] == nil {
|
||||
c.m[node] = map[string]string{}
|
||||
}
|
||||
c.m[node][engine] = variant
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestAnEngineBuildIsChosenForANodeAndTheNodeIsTold(t *testing.T) {
|
||||
a, ctl := setup(t, testModels(), gpuNode("gpu07", 8), inventory.Node{ID: "cpu01", State: inventory.Ready})
|
||||
ch := control.NewEngineChoices(t.Context(), &choiceStore{m: map[string]map[string]string{}}, slog.New(slog.DiscardHandler))
|
||||
var told []string
|
||||
ch.OnChange(func(node string) error { told = append(told, node); return nil })
|
||||
ctl.SetEngineChoices(ch)
|
||||
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"cu129"}`, nil); code != http.StatusOK {
|
||||
t.Fatalf("status %d", code)
|
||||
}
|
||||
if len(told) != 1 || told[0] != "gpu07" {
|
||||
t.Errorf("the node was not told: %v", told)
|
||||
}
|
||||
var nodes []NodeView
|
||||
call(t, a, http.MethodGet, "/api/v1/nodes", "", &nodes)
|
||||
for _, n := range nodes {
|
||||
switch n.ID {
|
||||
case "gpu07":
|
||||
if n.EngineBuilds["vllm"] != "cu129" {
|
||||
t.Errorf("gpu07 shows %v", n.EngineBuilds)
|
||||
}
|
||||
case "cpu01":
|
||||
if n.EngineBuilds != nil {
|
||||
t.Errorf("a node with no GPU has no build to choose: %v", n.EngineBuilds)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{`{"engine":"vllm","build":"cu200"}`, `{"engine":"llamacpp","build":"cu129"}`} {
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", bad, nil); code != http.StatusBadRequest {
|
||||
t.Errorf("%s: status %d", bad, code)
|
||||
}
|
||||
}
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"auto"}`, nil); code != http.StatusOK ||
|
||||
ch.For("gpu07")["vllm"] != "" {
|
||||
t.Errorf("auto should leave it to the node again: %d %v", code, ch.For("gpu07"))
|
||||
}
|
||||
}
|
||||
@@ -283,3 +283,36 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
|
||||
t.Errorf("the agent's refusal was not passed on: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
|
||||
// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
|
||||
// was kept out of placement. ComfyUI let go of it and the card stayed
|
||||
// out, saying "5014 MiB already allocated by another process", until the
|
||||
// agent was restarted.
|
||||
s := newTestService()
|
||||
nodeWith(s, inventory.Node{
|
||||
ID: "n1", State: inventory.Ready,
|
||||
GPUs: []discovery.GPU{{Index: 0, Health: discovery.GPUHealth{Problems: []string{
|
||||
discovery.ForeignMemoryProblem(5014), "2 uncorrected ECC errors"}}}},
|
||||
})
|
||||
|
||||
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{330}})
|
||||
n, _ := s.inv.Get("n1")
|
||||
if got := n.GPUs[0].Health.Problems; len(got) != 1 || got[0] != "2 uncorrected ECC errors" {
|
||||
t.Fatalf("problems are %q: the tenant left, and only the ECC errors should remain", got)
|
||||
}
|
||||
|
||||
// And the other way: a tenant arriving after registration.
|
||||
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{12000}})
|
||||
n, _ = s.inv.Get("n1")
|
||||
if got := n.GPUs[0].Health.Problems; len(got) != 2 || got[1] != discovery.ForeignMemoryProblem(12000) {
|
||||
t.Fatalf("problems are %q, want the new tenant's 12000 MiB", got)
|
||||
}
|
||||
|
||||
// An agent that sends no reading changes nothing.
|
||||
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuMemoryUsedMb: []int32{100}})
|
||||
n, _ = s.inv.Get("n1")
|
||||
if len(n.GPUs[0].Health.Problems) != 2 {
|
||||
t.Errorf("a heartbeat with no foreign reading cleared the card: %q", n.GPUs[0].Health.Problems)
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
|
||||
@@ -25,3 +26,57 @@ func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
|
||||
t.Error("uv, a tool for the others, is offered as an engine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetEnginesFor(func(n inventory.Node) map[string]string {
|
||||
if n.ID == "volta" {
|
||||
return map[string]string{"llamacpp": "b1", "vllm": "0.25.1"}
|
||||
}
|
||||
return map[string]string{"llamacpp": "b1", "vllm": "0.30.0"}
|
||||
})
|
||||
reg := &llmgwv1.Register{InstallableEngines: []string{"llamacpp", "vllm"}}
|
||||
want := func(p *llmgwv1.Provision) map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, e := range p.GetEngines() {
|
||||
out[e.GetEngine()] = e.GetVersion()
|
||||
}
|
||||
return out
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, installable(reg))); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("the Volta node: %v", got)
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, installable(reg))); got["vllm"] != "0.30.0" {
|
||||
t.Errorf("the Hopper node: %v", got)
|
||||
}
|
||||
// A node that disabled vLLM does not offer it, and is not asked for it.
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, []string{"llamacpp"})); got["vllm"] != "" {
|
||||
t.Errorf("asked a node for an engine it does not install: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildChosenForANodeIsWhatItIsTold(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetEngines(map[string]string{"llamacpp": "b1", "vllm": "0.30.0"})
|
||||
s.SetEngineChoices(func(node string) map[string]string {
|
||||
return map[string]map[string]string{"gpu07": {"vllm": "cu129"}, "gpu08": {"vllm": "none"}}[node]
|
||||
})
|
||||
can := []string{"llamacpp", "vllm"}
|
||||
variant := func(node string) (string, bool) {
|
||||
for _, e := range s.provisionFor(inventory.Node{ID: node}, can).GetEngines() {
|
||||
if e.GetEngine() == "vllm" {
|
||||
return e.GetVariant(), true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
if v, ok := variant("gpu07"); !ok || v != "cu129" {
|
||||
t.Errorf("gpu07: %q, %v", v, ok)
|
||||
}
|
||||
if _, ok := variant("gpu08"); ok {
|
||||
t.Error("gpu08 was asked for a vLLM it was chosen to run none of")
|
||||
}
|
||||
if v, ok := variant("gpu09"); !ok || v != "" {
|
||||
t.Errorf("gpu09, left to its hardware: %q, %v", v, ok)
|
||||
}
|
||||
}
|
||||
@@ -53,8 +53,13 @@ type Service struct {
|
||||
mu sync.RWMutex
|
||||
conns map[string]*conn
|
||||
|
||||
// engines is what every node is told to install, set once at start.
|
||||
engines map[string]string
|
||||
// enginesFor is what a node is told to install, set once at start: the
|
||||
// cluster's engines, with any override that picks the node laid over.
|
||||
enginesFor func(node inventory.Node) map[string]string
|
||||
// choices are the engine builds chosen per node, and installableBy what
|
||||
// each connected node said it can install, for telling it again.
|
||||
choices func(nodeID string) map[string]string
|
||||
installableBy map[string][]string
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -68,7 +73,37 @@ type Service struct {
|
||||
}
|
||||
|
||||
// SetEngines sets the engines every node installs, by name and version.
|
||||
func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
|
||||
func (s *Service) SetEngines(e map[string]string) {
|
||||
e = maps.Clone(e)
|
||||
s.enginesFor = func(inventory.Node) map[string]string { return e }
|
||||
}
|
||||
|
||||
// SetEnginesFor sets what each node installs, decided per node: a cluster
|
||||
// of two GPU generations needs two vLLM builds, or vLLM on only one.
|
||||
func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
|
||||
|
||||
// SetEngineChoices sets where the builds chosen for each node come from: by
|
||||
// engine, a build ("cu129") or "none". An engine not named is left to the
|
||||
// node's hardware.
|
||||
func (s *Service) SetEngineChoices(f func(nodeID string) map[string]string) { s.choices = f }
|
||||
|
||||
// Reprovision tells a connected node again what to install, after a choice
|
||||
// for it changed; one not connected is told when it next connects.
|
||||
func (s *Service) Reprovision(nodeID string) error {
|
||||
if s.enginesFor == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.RLock()
|
||||
can, ok := s.installableBy[nodeID]
|
||||
s.mu.RUnlock()
|
||||
node, known := s.inv.Get(nodeID)
|
||||
if !ok || !known {
|
||||
return nil
|
||||
}
|
||||
return s.send(nodeID, &llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: s.provisionFor(node, can)},
|
||||
})
|
||||
}
|
||||
|
||||
// conn is one agent's open stream.
|
||||
type conn struct {
|
||||
@@ -179,15 +214,15 @@ func (s *Service) Connect(
|
||||
// What this node should have installed. Sent on every registration, so a
|
||||
// node rebuilt from nothing is told the same as one that has been here a
|
||||
// month, and a changed version reaches every node on its next connect.
|
||||
if s.engines != nil {
|
||||
p := &llmgwv1.Provision{}
|
||||
if s.enginesFor != nil {
|
||||
can := installable(reg)
|
||||
for _, e := range slices.Sorted(maps.Keys(s.engines)) {
|
||||
if !slices.Contains(can, e) {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
|
||||
s.mu.Lock()
|
||||
if s.installableBy == nil {
|
||||
s.installableBy = map[string][]string{}
|
||||
}
|
||||
s.installableBy[nodeID] = can
|
||||
s.mu.Unlock()
|
||||
p := s.provisionFor(node, can)
|
||||
if err := stream.Send(&llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
|
||||
}); err != nil {
|
||||
@@ -342,11 +377,12 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
|
||||
// a reconnect.
|
||||
func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
used := hb.GetGpuMemoryUsedMb()
|
||||
foreign := hb.GetGpuForeignMemoryMb()
|
||||
problems := hb.GetNewProblems()
|
||||
storage := hb.GetStorage()
|
||||
memMB := int(hb.GetMemoryTotalMb())
|
||||
cpus := int(hb.GetCpuCount())
|
||||
if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
|
||||
if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 &&
|
||||
!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
|
||||
!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
|
||||
!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
|
||||
@@ -362,6 +398,20 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
n.GPUs[i].MemoryUsedMB = int(used[i])
|
||||
}
|
||||
}
|
||||
// Another tenant on a card, judged afresh: the reading at registration is
|
||||
// replaced, both ways. Without this a card that had something on it when
|
||||
// its agent started stayed out of placement after it left, and said so,
|
||||
// until the agent was restarted. Only a reading for every card replaces
|
||||
// it; an agent that sends none, or could not measure, leaves it standing.
|
||||
if len(foreign) == len(n.GPUs) {
|
||||
for i := range n.GPUs {
|
||||
ps := slices.DeleteFunc(slices.Clone(n.GPUs[i].Health.Problems), discovery.IsForeignMemoryProblem)
|
||||
if p := discovery.ForeignMemoryProblem(int(foreign[i])); p != "" {
|
||||
ps = append(ps, p)
|
||||
}
|
||||
n.GPUs[i].Health.Problems = ps
|
||||
}
|
||||
}
|
||||
if len(problems) > 0 {
|
||||
// Problems are reported per node, not per GPU, so attach them to the
|
||||
// node's first GPU rather than guessing which one is affected. Any
|
||||
@@ -550,6 +600,10 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
|
||||
out.GPUs = append(out.GPUs, inventory.GPUSensor{
|
||||
Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
|
||||
PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
|
||||
UtilPercent: g.GetUtilizationPercent(), MemUtilPercent: g.GetMemoryUtilizationPercent(),
|
||||
MemoryUsedMB: g.GetMemoryUsedMb(), MemoryTotalMB: g.GetMemoryTotalMb(),
|
||||
SMClockMHz: g.GetSmClockMhz(), ECCUncorrected: g.GetEccUncorrected(),
|
||||
Wear: wearFrom(g),
|
||||
})
|
||||
}
|
||||
return out
|
||||
@@ -853,6 +907,24 @@ func rate(before, now uint64, secs float64) float64 {
|
||||
return float64(now-before) / secs
|
||||
}
|
||||
|
||||
// provisionFor is what a node is told to install: the engines decided for
|
||||
// it, those it can install.
|
||||
func (s *Service) provisionFor(node inventory.Node, can []string) *llmgwv1.Provision {
|
||||
p := &llmgwv1.Provision{}
|
||||
wants := s.enginesFor(node)
|
||||
var chosen map[string]string
|
||||
if s.choices != nil {
|
||||
chosen = s.choices(node.ID)
|
||||
}
|
||||
for _, e := range slices.Sorted(maps.Keys(wants)) {
|
||||
if !slices.Contains(can, e) || chosen[e] == "none" {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e], Variant: chosen[e]})
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// installable are the engines a node can be asked to install. An agent older
|
||||
// than the list said nothing, and could install llama.cpp alone: asked for
|
||||
// vLLM too, it reported it unsupported, and a cluster half way through an
|
||||
@@ -908,3 +980,22 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
|
||||
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
|
||||
}
|
||||
}
|
||||
|
||||
// wearFrom is a card's wear, when the agent read it.
|
||||
func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
|
||||
if !g.GetHasWear() {
|
||||
return nil
|
||||
}
|
||||
w := &inventory.GPUWear{
|
||||
PCIeReplays: g.GetPcieReplays(), RetiredSBE: g.GetRetiredPagesSbe(), RetiredDBE: g.GetRetiredPagesDbe(),
|
||||
RetirePending: g.GetRetirementPending(), RemappedCorrectable: g.GetRemappedRowsCorrectable(),
|
||||
RemappedUncorrectable: g.GetRemappedRowsUncorrectable(), RemapPending: g.GetRemapPending(),
|
||||
RemapFailure: g.GetRemapFailure(), NVLinkCRC: g.GetNvlinkCrcErrors(), NVLinkReplay: g.GetNvlinkReplayErrors(),
|
||||
NVLinkRecovery: g.GetNvlinkRecoveryErrors(), XIDs: g.GetXidErrors(), LastXID: int(g.GetLastXid()),
|
||||
LastXIDText: g.GetLastXidText(),
|
||||
}
|
||||
if u := g.GetLastXidUnix(); u > 0 {
|
||||
w.LastXIDAt = time.Unix(u, 0)
|
||||
}
|
||||
return w
|
||||
}
|
||||
@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
|
||||
if p.Nodes <= 0 {
|
||||
p.Nodes = 1
|
||||
}
|
||||
// tp on a llama.cpp GPU profile, written before gpus_per_node meant
|
||||
// anything for it, is read as the width it was meant as: N cards of
|
||||
// one node. Refusing it would have dropped a catalog's model on the
|
||||
// upgrade that made the width work. Only both given, disagreeing, is
|
||||
// refused below, as ambiguous.
|
||||
if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
|
||||
if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
|
||||
p.GPUsPerNode, p.TP, p.PP = n, 1, 0
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case p.CPUOnly():
|
||||
// Nothing to divide up: this profile asks for no cards at all.
|
||||
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
|
||||
p.Weight = 1
|
||||
}
|
||||
}
|
||||
// A Hugging Face repository page for a model vLLM or SGLang runs is the
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -333,6 +350,15 @@ func (m Model) validate() error {
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -381,6 +407,22 @@ func (m Model) validate() error {
|
||||
}
|
||||
continue
|
||||
}
|
||||
// llama.cpp on GPUs splits a model across one node's cards by layer,
|
||||
// and how many is gpus_per_node. Not tp: llama.cpp has no tensor
|
||||
// parallelism, and a tp it was given was a width nothing placed — the
|
||||
// replica stayed unplaced, or ran on one card while four were meant.
|
||||
if p.Engine == "llamacpp" {
|
||||
switch {
|
||||
case p.Nodes > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
m.Name, p.Name, p.Nodes)
|
||||
case p.TP > 1 || p.PP > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
|
||||
"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
|
||||
max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
|
||||
}
|
||||
continue
|
||||
}
|
||||
// The arithmetic has to add up, or the engine is handed a shard spec it
|
||||
// cannot satisfy and fails minutes later during model loading.
|
||||
if want := p.GPUsPerNode * p.Nodes; want != p.TP*max(p.PP, 1) {
|
||||
@@ -704,3 +746,21 @@ func (m Model) engineAdapters() []engines.Adapter {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// repositoryPage reports whether source is a Hugging Face repository's page
|
||||
// (https://huggingface.co/org/repo, or one of its tree views) rather than a
|
||||
// file under /resolve/, and the repository's id.
|
||||
func repositoryPage(source string) (string, bool) {
|
||||
for _, host := range []string{"https://huggingface.co/", "http://huggingface.co/", "https://hf.co/"} {
|
||||
rest, ok := strings.CutPrefix(source, host)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
parts := strings.Split(strings.Trim(rest, "/"), "/")
|
||||
if len(parts) < 2 || (len(parts) > 2 && parts[2] == "resolve") {
|
||||
return "", false
|
||||
}
|
||||
return parts[0] + "/" + parts[1], true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
@@ -321,3 +321,67 @@ func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
|
||||
t.Errorf("ordinary tuning refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
|
||||
four := model("big", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", GPUsPerNode: 4})
|
||||
c, err := New([]Model{four})
|
||||
if err != nil {
|
||||
t.Fatalf("gpus_per_node 4 on llama.cpp: %v", err)
|
||||
}
|
||||
m, _ := c.Resolve("big")
|
||||
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
|
||||
t.Errorf("placement was not asked for four GPUs: %+v", pl)
|
||||
}
|
||||
// tp, written before gpus_per_node meant anything for llama.cpp, is read
|
||||
// as the same width rather than refused: refusing dropped the model on
|
||||
// the upgrade that made the width work.
|
||||
old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
|
||||
c, err = New([]Model{old})
|
||||
if err != nil {
|
||||
t.Fatalf("tp 4 on llama.cpp: %v", err)
|
||||
}
|
||||
got, _ := c.Resolve("old")
|
||||
if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
|
||||
t.Errorf("tp 4 should place on four GPUs: %+v", p)
|
||||
}
|
||||
// Both given and disagreeing is ambiguous, and refused; so is more than
|
||||
// one node.
|
||||
for _, p := range []Profile{
|
||||
{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
|
||||
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
|
||||
} {
|
||||
if _, err := New([]Model{model("x", p)}); err == nil {
|
||||
t.Errorf("%s: accepted", p.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
|
||||
// The field: an AWQ model's page given as a vLLM source, fetched as one
|
||||
// file of HTML. vLLM downloads a repository id itself.
|
||||
for _, src := range []string{
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
|
||||
} {
|
||||
m := model("awq")
|
||||
m.Source = src
|
||||
c, err := New([]Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", src, err)
|
||||
}
|
||||
if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
|
||||
t.Errorf("%s became %q", src, got.Source)
|
||||
}
|
||||
}
|
||||
// llama.cpp needs a file, which a page is not.
|
||||
g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
|
||||
g.Source = "https://huggingface.co/Qwen/X-GGUF"
|
||||
if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
|
||||
t.Errorf("a page for llama.cpp: %v", err)
|
||||
}
|
||||
f := model("file")
|
||||
f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{f}); err != nil {
|
||||
t.Errorf("a file under /resolve/ is a source: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -75,15 +75,18 @@ func DefaultConfig() Config {
|
||||
|
||||
// Controller owns the control plane's state.
|
||||
type Controller struct {
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
lead *Leadership
|
||||
disp reconciler.Dispatcher
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
// engineChoices are the engine builds chosen per node; nil without a
|
||||
// store to keep them in.
|
||||
engineChoices *EngineChoices
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
lead *Leadership
|
||||
disp reconciler.Dispatcher
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
|
||||
mu sync.Mutex
|
||||
last string // fingerprint of the last published list
|
||||
@@ -408,3 +411,23 @@ func EngineKeys(secret []byte) func(replicaID string) string {
|
||||
|
||||
// QuotaUse is every capacity quota beside what its scope holds.
|
||||
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
|
||||
|
||||
// SetEngineChoices gives the controller the engine builds chosen per node.
|
||||
func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
|
||||
|
||||
// EngineChoices are the engine builds chosen per node, or nil.
|
||||
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
|
||||
|
||||
// ModelOwners maps each model to the part of the tenancy tree its capacity
|
||||
// counts against; a model owned by nobody is not in it.
|
||||
func (c *Controller) ModelOwners() map[string]string {
|
||||
out := map[string]string{}
|
||||
if cat := c.Catalog(); cat != nil {
|
||||
for _, m := range cat.Models() {
|
||||
if m.Owner != "" {
|
||||
out[m.Name] = m.Owner
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
)
|
||||
|
||||
// EngineChoiceStore is where the engine builds chosen per node are kept.
|
||||
type EngineChoiceStore interface {
|
||||
NodeEngines(ctx context.Context) (map[string]map[string]string, error)
|
||||
SetNodeEngine(ctx context.Context, node, engine, variant, by string) error
|
||||
}
|
||||
|
||||
// EngineChoices are the engine builds chosen for nodes, rather than left to
|
||||
// their hardware: vLLM's CUDA 12.9 build on a node its CUDA 13 build would
|
||||
// also install on, or no vLLM at all. Kept in the store, held here for the
|
||||
// agent service to ask on every registration, and pushed to a node the
|
||||
// moment its choice changes.
|
||||
type EngineChoices struct {
|
||||
st EngineChoiceStore
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.RWMutex
|
||||
byNode map[string]map[string]string
|
||||
notify func(node string) error
|
||||
}
|
||||
|
||||
// NewEngineChoices reads what was chosen before.
|
||||
func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logger) *EngineChoices {
|
||||
c := &EngineChoices{st: st, log: log, byNode: map[string]map[string]string{}}
|
||||
if got, err := st.NodeEngines(ctx); err != nil {
|
||||
log.Error("could not read the engine builds chosen per node; every node installs what its hardware runs", "err", err)
|
||||
} else {
|
||||
c.byNode = got
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// OnChange is called with a node whose choice changed, to tell it.
|
||||
func (c *EngineChoices) OnChange(f func(node string) error) {
|
||||
c.mu.Lock()
|
||||
c.notify = f
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// For is what was chosen for one node, by engine: a build, or "none".
|
||||
func (c *EngineChoices) For(node string) map[string]string {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
return maps.Clone(c.byNode[node])
|
||||
}
|
||||
|
||||
// EngineBuilds are what can be chosen for each engine that has a choice:
|
||||
// "auto" is the node's hardware deciding, "none" no engine at all.
|
||||
func EngineBuilds(engine string) []string {
|
||||
switch engine {
|
||||
case "vllm":
|
||||
return append(append([]string{"auto"}, provision.VLLMVariants...), "none")
|
||||
case "sglang":
|
||||
return []string{"auto", "none"}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Set chooses an engine's build for a node, keeps it, and tells the node.
|
||||
func (c *EngineChoices) Set(ctx context.Context, node, engine, build, by string) error {
|
||||
builds := EngineBuilds(engine)
|
||||
if builds == nil {
|
||||
return fmt.Errorf("%s has no builds to choose between", engine)
|
||||
}
|
||||
if !slices.Contains(builds, build) {
|
||||
return fmt.Errorf("%s has no build %q; choose %s", engine, build, strings.Join(builds, ", "))
|
||||
}
|
||||
variant := build
|
||||
if build == "auto" {
|
||||
variant = ""
|
||||
}
|
||||
if err := c.st.SetNodeEngine(ctx, node, engine, variant, by); err != nil {
|
||||
return err
|
||||
}
|
||||
c.mu.Lock()
|
||||
if variant == "" {
|
||||
delete(c.byNode[node], engine)
|
||||
} else {
|
||||
if c.byNode[node] == nil {
|
||||
c.byNode[node] = map[string]string{}
|
||||
}
|
||||
c.byNode[node][engine] = variant
|
||||
}
|
||||
notify := c.notify
|
||||
c.mu.Unlock()
|
||||
c.log.Info("engine build chosen for a node", "node", node, "engine", engine, "build", build, "by", by)
|
||||
if notify != nil {
|
||||
if err := notify(node); err != nil {
|
||||
// Kept and applied on its next connection; said, since a node that
|
||||
// is not connected is the usual reason.
|
||||
c.log.Warn("the node was not told yet; it is when it next connects", "node", node, "err", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -73,6 +73,18 @@ type Collector struct {
|
||||
gpuMemoryTotal *prometheus.Desc
|
||||
gpuMemoryUsed *prometheus.Desc
|
||||
gpuUtil *prometheus.Desc
|
||||
gpuAssigned *prometheus.Desc
|
||||
gpuPCIeReplay *prometheus.Desc
|
||||
gpuRetired *prometheus.Desc
|
||||
gpuRetirePend *prometheus.Desc
|
||||
gpuRemapped *prometheus.Desc
|
||||
gpuRemapPend *prometheus.Desc
|
||||
gpuRemapFail *prometheus.Desc
|
||||
gpuNVLinkErr *prometheus.Desc
|
||||
gpuXID *prometheus.Desc
|
||||
gpuLastXID *prometheus.Desc
|
||||
gpuMemUtil *prometheus.Desc
|
||||
gpuSMClock *prometheus.Desc
|
||||
gpuTemp *prometheus.Desc
|
||||
gpuMemTemp *prometheus.Desc
|
||||
gpuPower *prometheus.Desc
|
||||
@@ -149,6 +161,33 @@ func New(src Source) *Collector {
|
||||
"How busy a GPU was over the last sample period. Absent when the driver does not "+
|
||||
"report it, which is not the same as idle. Memory used says a model is loaded; "+
|
||||
"this says whether it is doing anything.", gpu, nil),
|
||||
gpuPCIeReplay: prometheus.NewDesc("llmgw_gpu_pcie_replays",
|
||||
"PCIe link retries on a GPU's link since the driver loaded; one that keeps climbing is a riser, slot or cable.", gpu, nil),
|
||||
gpuRetired: prometheus.NewDesc("llmgw_gpu_retired_pages",
|
||||
"Memory pages retired, by cause (sbe, dbe), on GPUs before Ampere.", append(gpu, "cause"), nil),
|
||||
gpuRetirePend: prometheus.NewDesc("llmgw_gpu_retirement_pending",
|
||||
"1 when retired pages wait for a GPU reset to take effect.", gpu, nil),
|
||||
gpuRemapped: prometheus.NewDesc("llmgw_gpu_remapped_rows",
|
||||
"Memory rows remapped, by kind (correctable, uncorrectable), on Ampere and later.", append(gpu, "kind"), nil),
|
||||
gpuRemapPend: prometheus.NewDesc("llmgw_gpu_remap_pending",
|
||||
"1 when remapped rows wait for a GPU reset to take effect.", gpu, nil),
|
||||
gpuRemapFail: prometheus.NewDesc("llmgw_gpu_remap_failure",
|
||||
"1 when a GPU has run out of spare memory rows: it needs replacing.", gpu, nil),
|
||||
gpuNVLinkErr: prometheus.NewDesc("llmgw_gpu_nvlink_errors",
|
||||
"NVLink errors summed over a GPU's links, by kind (crc, replay, recovery).", append(gpu, "kind"), nil),
|
||||
gpuXID: prometheus.NewDesc("llmgw_gpu_xid_errors",
|
||||
"The driver's XID error events for a GPU since its node's agent started.", gpu, nil),
|
||||
gpuLastXID: prometheus.NewDesc("llmgw_gpu_last_xid",
|
||||
"The code of a GPU's last XID error since its agent started; absent when there was none.", gpu, nil),
|
||||
gpuAssigned: prometheus.NewDesc("llmgw_gpu_assigned",
|
||||
"1 for each GPU a replica holds: which model, which replica, and whose quota it counts against. "+
|
||||
"Joined on node and gpu with the other llmgw_gpu_ series, it says what each model's cards are doing.",
|
||||
[]string{"node", "gpu", "model", "replica", "owner"}, nil),
|
||||
gpuMemUtil: prometheus.NewDesc("llmgw_gpu_memory_utilization_percent",
|
||||
"How much of the sample period a GPU's memory was being read or written: "+
|
||||
"decode is bound by it, so a card busy here and idle above is waiting on memory.", gpu, nil),
|
||||
gpuSMClock: prometheus.NewDesc("llmgw_gpu_sm_clock_mhz",
|
||||
"A GPU's streaming multiprocessor clock now; one held low under load is the card being throttled.", gpu, nil),
|
||||
gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
|
||||
"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
|
||||
gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
|
||||
@@ -271,7 +310,9 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
|
||||
c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
|
||||
c.gpuNVLJoined, c.imexUp,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuPCIeReplay, c.gpuRetired, c.gpuRetirePend, c.gpuRemapped, c.gpuRemapPend, c.gpuRemapFail,
|
||||
c.gpuNVLinkErr, c.gpuXID, c.gpuLastXID,
|
||||
c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
|
||||
c.gpuNVLink, c.gpuECC,
|
||||
c.replicaState, c.replicaReady, c.replicaIdle,
|
||||
@@ -287,6 +328,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
now := time.Now()
|
||||
c.collectQuotas(ch)
|
||||
c.collectAssignments(ch)
|
||||
|
||||
loads := map[string]*inventory.Load{}
|
||||
for _, n := range c.src.Nodes() {
|
||||
@@ -388,20 +430,28 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
byClass[class]++
|
||||
labels := []string{n.ID, strconv.Itoa(g.Index), class}
|
||||
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
|
||||
float64(g.MemoryTotalMB)*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
|
||||
float64(g.MemoryUsedMB)*1024*1024, labels...)
|
||||
if g.UtilizationPercent > 0 {
|
||||
// Only when reported. A driver too old to give it would
|
||||
// otherwise show every card at a flat zero, which reads as an
|
||||
// idle fleet rather than a missing measurement.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
|
||||
float64(g.UtilizationPercent), labels...)
|
||||
s, fresh := sensorFor(n.Load, g.Index, now)
|
||||
if fresh && s.Busy() {
|
||||
// What the card is doing now, sampled every heartbeat.
|
||||
c.gpuBusy(ch, s, labels)
|
||||
} else {
|
||||
// An agent too old to sample it: what it read at
|
||||
// registration, which is all there is.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
|
||||
float64(g.MemoryTotalMB)*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
|
||||
float64(g.MemoryUsedMB)*1024*1024, labels...)
|
||||
if g.UtilizationPercent > 0 {
|
||||
// Only when reported. A driver too old to give it would
|
||||
// otherwise show every card at a flat zero, which reads as
|
||||
// an idle fleet rather than a missing measurement.
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
|
||||
float64(g.UtilizationPercent), labels...)
|
||||
}
|
||||
}
|
||||
// The sensors' fresh reading when there is one; the temperature
|
||||
// read at registration otherwise, which is all an older agent sends.
|
||||
if s, ok := sensorFor(n.Load, g.Index, now); ok {
|
||||
if fresh {
|
||||
c.gpuSensor(ch, s, labels)
|
||||
} else if g.Health.TemperatureC > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
|
||||
@@ -414,8 +464,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuHealthy, prometheus.GaugeValue, healthy, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuNVLink, prometheus.GaugeValue,
|
||||
float64(len(g.NVLinkPeers)), labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue,
|
||||
float64(g.Health.ECCErrors), labels...)
|
||||
ecc := float64(g.Health.ECCErrors)
|
||||
if fresh && s.Busy() && s.ECCUncorrected >= 0 {
|
||||
ecc = s.ECCUncorrected
|
||||
}
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue, ecc, labels...)
|
||||
}
|
||||
for class, n2 := range byClass {
|
||||
ch <- prometheus.MustNewConstMetric(c.nodeGPUs, prometheus.GaugeValue,
|
||||
@@ -521,6 +574,12 @@ func b2f(b bool) float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
// OwnerSource says whose quota each model counts against, for labelling
|
||||
// what its replicas hold. Asked for by type assertion.
|
||||
type OwnerSource interface {
|
||||
ModelOwners() map[string]string
|
||||
}
|
||||
|
||||
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
|
||||
// assertion, like HealthSource.
|
||||
type QuotaSource interface {
|
||||
@@ -569,6 +628,23 @@ func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor
|
||||
return inventory.GPUSensor{}, false
|
||||
}
|
||||
|
||||
// gpuBusy is how busy a card is, from a sample that carries it.
|
||||
func (c *Collector) gpuBusy(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue, s.MemoryTotalMB*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue, s.MemoryUsedMB*1024*1024, labels...)
|
||||
// Zero is a real reading here — an idle card — and only -1, the driver
|
||||
// not saying, leaves the series out.
|
||||
if s.UtilPercent >= 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue, s.UtilPercent, labels...)
|
||||
}
|
||||
if s.MemUtilPercent >= 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemUtil, prometheus.GaugeValue, s.MemUtilPercent, labels...)
|
||||
}
|
||||
if s.SMClockMHz > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuSMClock, prometheus.GaugeValue, s.SMClockMHz, labels...)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
|
||||
for _, v := range []struct {
|
||||
d *prometheus.Desc
|
||||
@@ -581,4 +657,61 @@ func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor
|
||||
for _, reason := range s.Throttle {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
|
||||
}
|
||||
if w := s.Wear; w != nil {
|
||||
c.gpuWear(ch, w, labels)
|
||||
}
|
||||
}
|
||||
|
||||
// gpuWear is what wears a card out. A count the card does not keep (-1) is
|
||||
// left out, not published as zero.
|
||||
func (c *Collector) gpuWear(ch chan<- prometheus.Metric, w *inventory.GPUWear, labels []string) {
|
||||
gauge := func(d *prometheus.Desc, v float64, extra ...string) {
|
||||
ch <- prometheus.MustNewConstMetric(d, prometheus.GaugeValue, v, append(append([]string{}, labels...), extra...)...)
|
||||
}
|
||||
flag := func(b bool) float64 {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
if w.PCIeReplays >= 0 {
|
||||
gauge(c.gpuPCIeReplay, float64(w.PCIeReplays))
|
||||
}
|
||||
if w.RetiredSBE >= 0 || w.RetiredDBE >= 0 {
|
||||
gauge(c.gpuRetired, float64(max(w.RetiredSBE, 0)), "sbe")
|
||||
gauge(c.gpuRetired, float64(max(w.RetiredDBE, 0)), "dbe")
|
||||
gauge(c.gpuRetirePend, flag(w.RetirePending))
|
||||
}
|
||||
if w.RemappedCorrectable >= 0 || w.RemappedUncorrectable >= 0 {
|
||||
gauge(c.gpuRemapped, float64(max(w.RemappedCorrectable, 0)), "correctable")
|
||||
gauge(c.gpuRemapped, float64(max(w.RemappedUncorrectable, 0)), "uncorrectable")
|
||||
gauge(c.gpuRemapPend, flag(w.RemapPending))
|
||||
gauge(c.gpuRemapFail, flag(w.RemapFailure))
|
||||
}
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkCRC), "crc")
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkReplay), "replay")
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkRecovery), "recovery")
|
||||
gauge(c.gpuXID, float64(w.XIDs))
|
||||
if w.XIDs > 0 {
|
||||
gauge(c.gpuLastXID, float64(w.LastXID))
|
||||
}
|
||||
}
|
||||
|
||||
// collectAssignments says which replica holds each GPU.
|
||||
func (c *Collector) collectAssignments(ch chan<- prometheus.Metric) {
|
||||
var owners map[string]string
|
||||
if o, ok := c.src.(OwnerSource); ok {
|
||||
owners = o.ModelOwners()
|
||||
}
|
||||
for _, r := range c.src.Replicas() {
|
||||
if !r.State.Counts() {
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
for _, g := range sh.GPUIndices {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuAssigned, prometheus.GaugeValue, 1,
|
||||
sh.NodeID, strconv.Itoa(g), r.Model, r.ID, owners[r.Model])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
@@ -39,3 +40,90 @@ func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHowBusyAGPUIsComesFromTheLatestSampleNotRegistration(t *testing.T) {
|
||||
// Discovery read 87% and 70 GB used when the agent came up; the card has
|
||||
// been idle since. The sample says so, zero included.
|
||||
reg := discovery.GPU{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
|
||||
MemoryUsedMB: 70000, UtilizationPercent: 87}
|
||||
n := inventory.Node{ID: "g1", State: inventory.Ready, GPUs: []discovery.GPU{reg},
|
||||
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 35,
|
||||
UtilPercent: 0, MemUtilPercent: 0, MemoryUsedMB: 512, MemoryTotalMB: 81559,
|
||||
SMClockMHz: 345, ECCUncorrected: 2}}}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
|
||||
"llmgw_gpu_memory_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
|
||||
"llmgw_gpu_memory_used_bytes class=h100-80g gpu=0 node=g1 -> 536870912\n",
|
||||
"llmgw_gpu_sm_clock_mhz class=h100-80g gpu=0 node=g1 -> 345\n",
|
||||
"llmgw_gpu_ecc_errors class=h100-80g gpu=0 node=g1 -> 2\n",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s in:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "-> 87\n") {
|
||||
t.Error("the utilization read at registration was published over the sample's")
|
||||
}
|
||||
|
||||
// An agent too old to sample it: registration's figures, and no zero
|
||||
// standing in for a utilization the driver did not give.
|
||||
n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35}}
|
||||
out = gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
if !strings.Contains(out, "llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 87") {
|
||||
t.Errorf("an older agent's registration figure was dropped:\n%s", out)
|
||||
}
|
||||
n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35, UtilPercent: -1, MemUtilPercent: -1, MemoryTotalMB: 81559}}
|
||||
if out := gather(t, fakeSource{nodes: []inventory.Node{n}}); strings.Contains(out, "llmgw_gpu_utilization_percent ") {
|
||||
t.Errorf("a utilization the driver did not report was published:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// ownedSource adds owners to fakeSource.
|
||||
type ownedSource struct{ fakeSource }
|
||||
|
||||
func (ownedSource) ModelOwners() map[string]string { return map[string]string{"chat": "acme/ml"} }
|
||||
|
||||
func TestEachGPUAReplicaHoldsSaysWhoseItIs(t *testing.T) {
|
||||
src := ownedSource{fakeSource{replicas: []reconciler.Replica{
|
||||
{ID: "chat-1", Model: "chat", State: reconciler.Ready, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{0, 1}}}},
|
||||
{ID: "old-1", Model: "old", State: reconciler.Stopped, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}},
|
||||
}}}
|
||||
out := gather(t, src)
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_assigned gpu=0 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
|
||||
"llmgw_gpu_assigned gpu=1 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s in:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "replica=old-1") {
|
||||
t.Error("a stopped replica's GPU was counted as held")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGPUsWearIsPublishedAndWhatItDoesNotKeepIsLeftOut(t *testing.T) {
|
||||
n := inventory.Node{ID: "g1", State: inventory.Ready,
|
||||
GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559}},
|
||||
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 40, MemoryTotalMB: 81559,
|
||||
Wear: &inventory.GPUWear{PCIeReplays: 12, RetiredSBE: -1, RetiredDBE: -1, RemappedCorrectable: 2,
|
||||
RemappedUncorrectable: 1, RemapPending: true, NVLinkCRC: 7, XIDs: 3, LastXID: 79}}}}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_pcie_replays class=h100-80g gpu=0 node=g1 -> 12\n",
|
||||
"llmgw_gpu_remapped_rows class=h100-80g gpu=0 kind=uncorrectable node=g1 -> 1\n",
|
||||
"llmgw_gpu_remap_pending class=h100-80g gpu=0 node=g1 -> 1\n",
|
||||
"llmgw_gpu_nvlink_errors class=h100-80g gpu=0 kind=crc node=g1 -> 7\n",
|
||||
"llmgw_gpu_xid_errors class=h100-80g gpu=0 node=g1 -> 3\n",
|
||||
"llmgw_gpu_last_xid class=h100-80g gpu=0 node=g1 -> 79\n",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s", want)
|
||||
}
|
||||
}
|
||||
// An H100 keeps no retired pages; none are published.
|
||||
if strings.Contains(out, "llmgw_gpu_retired_pages") {
|
||||
t.Errorf("a count the card does not keep was published:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@
|
||||
package inventory
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -319,8 +320,74 @@ type GPUSensor struct {
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerLimitW float64 `json:"power_limit_w,omitempty"`
|
||||
Throttle []string `json:"throttle,omitempty"`
|
||||
// How busy the card is, as of this sample; -1 is not reported. Zero
|
||||
// MemoryTotalMB is an agent too old to send these, whose figures are
|
||||
// the ones it read at registration.
|
||||
UtilPercent float64 `json:"util_percent,omitempty"`
|
||||
MemUtilPercent float64 `json:"mem_util_percent,omitempty"`
|
||||
MemoryUsedMB float64 `json:"memory_used_mb,omitempty"`
|
||||
MemoryTotalMB float64 `json:"memory_total_mb,omitempty"`
|
||||
SMClockMHz float64 `json:"sm_clock_mhz,omitempty"`
|
||||
ECCUncorrected float64 `json:"ecc_uncorrected,omitempty"`
|
||||
// Wear is what wears the card out, read every minute; nil from an agent
|
||||
// that does not read it.
|
||||
Wear *GPUWear `json:"wear,omitempty"`
|
||||
}
|
||||
|
||||
// GPUWear is a card's wear and failures. Counts of -1 are not reported for
|
||||
// it: a card before Ampere has no remapped rows, one after no retired pages.
|
||||
type GPUWear struct {
|
||||
PCIeReplays int64 `json:"pcie_replays"`
|
||||
RetiredSBE int64 `json:"retired_pages_sbe"`
|
||||
RetiredDBE int64 `json:"retired_pages_dbe"`
|
||||
RetirePending bool `json:"retire_pending,omitempty"`
|
||||
RemappedCorrectable int64 `json:"remapped_rows_correctable"`
|
||||
RemappedUncorrectable int64 `json:"remapped_rows_uncorrectable"`
|
||||
RemapPending bool `json:"remap_pending,omitempty"`
|
||||
RemapFailure bool `json:"remap_failure,omitempty"`
|
||||
NVLinkCRC int64 `json:"nvlink_crc_errors"`
|
||||
NVLinkReplay int64 `json:"nvlink_replay_errors"`
|
||||
NVLinkRecovery int64 `json:"nvlink_recovery_errors"`
|
||||
XIDs int64 `json:"xid_errors"`
|
||||
LastXID int `json:"last_xid,omitempty"`
|
||||
LastXIDText string `json:"last_xid_text,omitempty"`
|
||||
LastXIDAt time.Time `json:"last_xid_at,omitempty"`
|
||||
}
|
||||
|
||||
// Problems says, in a few words each, what is wrong with the card; empty
|
||||
// when nothing is.
|
||||
func (w *GPUWear) Problems() []string {
|
||||
if w == nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
if w.RemapFailure {
|
||||
out = append(out, "out of spare memory rows: replace the card")
|
||||
}
|
||||
if w.RemapPending {
|
||||
out = append(out, "memory rows remapped, waiting for a reset")
|
||||
}
|
||||
if w.RetirePending {
|
||||
out = append(out, "memory pages waiting for a reset to be retired")
|
||||
}
|
||||
if w.RemappedUncorrectable > 0 {
|
||||
out = append(out, fmt.Sprintf("%d rows remapped for uncorrectable errors", w.RemappedUncorrectable))
|
||||
}
|
||||
if w.RetiredDBE > 0 {
|
||||
out = append(out, fmt.Sprintf("%d pages retired for double-bit errors", w.RetiredDBE))
|
||||
}
|
||||
if w.XIDs > 0 {
|
||||
out = append(out, fmt.Sprintf("XID %d (%s)", w.LastXID, w.LastXIDText))
|
||||
}
|
||||
if w.NVLinkCRC > 0 || w.NVLinkReplay > 0 || w.NVLinkRecovery > 0 {
|
||||
out = append(out, fmt.Sprintf("NVLink errors: %d CRC, %d replay, %d recovery", w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Busy reports whether the sample carries how busy the card is.
|
||||
func (s GPUSensor) Busy() bool { return s.MemoryTotalMB > 0 }
|
||||
|
||||
// ShardLoad is one engine's use: cores busy, and resident memory.
|
||||
type ShardLoad struct {
|
||||
CPUCores float64
|
||||
|
||||
@@ -255,6 +255,11 @@ type Reconciler struct {
|
||||
// restarted and not yet begun. One is replaced at a time: replacing them
|
||||
// all at once would need the memory of every replica twice over.
|
||||
restartQueue map[string][]string
|
||||
// restartInPlace are the replicas asked to restart by stopping first.
|
||||
restartInPlace map[string]bool
|
||||
// restartRefused says, by replica, why a restart was given up rather
|
||||
// than leave the model without it.
|
||||
restartRefused map[string]string
|
||||
// parked holds models that failed ParkAfter times in a row.
|
||||
parked map[string]parkRecord
|
||||
// extra is how many replicas load has added above a model's base, and
|
||||
@@ -304,6 +309,8 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
|
||||
reservedPorts: map[int]struct{}{},
|
||||
wanted: map[string]time.Time{},
|
||||
restartQueue: map[string][]string{},
|
||||
restartInPlace: map[string]bool{},
|
||||
restartRefused: map[string]string{},
|
||||
parked: map[string]parkRecord{},
|
||||
extra: map[string]int{},
|
||||
busySince: map[string]time.Time{},
|
||||
@@ -729,6 +736,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
r.mu.Unlock()
|
||||
|
||||
desired := r.desiredFor(m)
|
||||
outcome := notTried
|
||||
switch {
|
||||
case len(liveIDs) < desired && isParked:
|
||||
// Not placed. Its failure is on every status the operator reads.
|
||||
@@ -742,7 +750,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// the real error in noise.
|
||||
return
|
||||
}
|
||||
r.scaleUp(ctx, m, avoid)
|
||||
outcome = r.scaleUp(ctx, m, avoid)
|
||||
case len(liveIDs) > desired:
|
||||
r.scaleDown(ctx, liveIDs[desired:])
|
||||
}
|
||||
@@ -781,7 +789,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
}
|
||||
}
|
||||
|
||||
r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
|
||||
r.retire(ctx, m, retiringIDs, readyLive, len(liveIDs), desired, outcome, isParked)
|
||||
}
|
||||
|
||||
// scaleUp places and starts one replica per pass.
|
||||
@@ -789,7 +797,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// One at a time on purpose: placement decisions depend on what is already
|
||||
// allocated, and starting several at once from the same inventory snapshot
|
||||
// would hand them the same GPUs.
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) scaleOutcome {
|
||||
profiles := r.orderedProfiles(m)
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
@@ -813,13 +821,13 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
// the pass that reaps it, and the next pass places this model — which
|
||||
// keeps the stop-then-start ordering in one place rather than two.
|
||||
if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
|
||||
return
|
||||
return makingRoom
|
||||
}
|
||||
if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
|
||||
return
|
||||
return makingRoom
|
||||
}
|
||||
r.notePlacementFailure(m.Name, res.Explain())
|
||||
return
|
||||
return noRoom
|
||||
}
|
||||
if why := r.overLimit(m, res.Placement); why != "" {
|
||||
// Recorded as the placement's answer, so the model reads as not
|
||||
@@ -829,13 +837,26 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
{Profile: res.Placement.Profile.Name, Reason: why}}}
|
||||
r.mu.Unlock()
|
||||
r.notePlacementFailure(m.Name, why)
|
||||
return
|
||||
return overLimit
|
||||
}
|
||||
if err := r.startReplica(ctx, m, res.Placement); err != nil {
|
||||
r.log.Error("starting replica failed", "model", m.Name, "err", err)
|
||||
return noRoom
|
||||
}
|
||||
return placed
|
||||
}
|
||||
|
||||
// scaleOutcome is what one scaleUp did.
|
||||
type scaleOutcome int
|
||||
|
||||
const (
|
||||
notTried scaleOutcome = iota // nothing was asked of placement this pass
|
||||
placed // a replica was started
|
||||
makingRoom // room is being made: a preemption, an idle replica stopping
|
||||
noRoom // placement found nowhere, and nothing is making room
|
||||
overLimit // there is room, but the owner's quota does not allow it
|
||||
)
|
||||
|
||||
func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *placement.Placement) error {
|
||||
prof, ok := m.ProfileByName(pl.Profile.Name)
|
||||
if !ok {
|
||||
|
||||
@@ -114,6 +114,17 @@ type harness struct {
|
||||
inv *inventory.Store
|
||||
disp *fakeDispatcher
|
||||
clock *clock
|
||||
live *catalog.Live
|
||||
}
|
||||
|
||||
// setModels replaces the catalog the reconciler works from, as an edit does.
|
||||
func (h *harness) setModels(t *testing.T, models ...catalog.Model) {
|
||||
t.Helper()
|
||||
cat, err := catalog.New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.live.Store(cat)
|
||||
}
|
||||
|
||||
func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness {
|
||||
@@ -135,7 +146,8 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
|
||||
cfg.UnreachableAfter = 15 * time.Second
|
||||
cfg.LostAfter = 60 * time.Second
|
||||
|
||||
r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
|
||||
live := catalog.NewLive(cat)
|
||||
r := New(cfg, live, inv, disp, slog.New(slog.DiscardHandler))
|
||||
r.SetClock(c.now)
|
||||
// Nodes were just registered, so their LastSeen is wall-clock now, which is
|
||||
// far from the fake clock's epoch. Re-stamp them onto the fake timeline.
|
||||
@@ -150,7 +162,7 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
|
||||
for _, n := range nodes {
|
||||
touchAt(inv, n.ID, c.now())
|
||||
}
|
||||
return &harness{r: r, inv: inv, disp: disp, clock: c}
|
||||
return &harness{r: r, inv: inv, disp: disp, clock: c, live: live}
|
||||
}
|
||||
|
||||
// touchAt rewrites a node's LastSeen onto the fake clock's timeline.
|
||||
@@ -1787,7 +1799,9 @@ func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
|
||||
|
||||
func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
// One node, room for one replica: the replacement can only start once the
|
||||
// old one has gone. Waiting forever would mean never restarting at all.
|
||||
// old one has gone. It is the only thing in the way, so it goes now —
|
||||
// waiting out a grace period first changed nothing but the wait, which
|
||||
// in the field read as a restart that never happened.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
@@ -1796,14 +1810,8 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, _ := replicaByID(h, old); rep.State != Ready {
|
||||
t.Fatalf("stopped before the grace period: %s", rep.State)
|
||||
}
|
||||
|
||||
h.advance(h.r.cfg.DrainGrace + time.Second)
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
|
||||
t.Fatal("still serving after the grace period with no replacement possible")
|
||||
t.Fatal("still serving with no replacement possible beside it")
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
h.r.Reconcile(t.Context())
|
||||
@@ -1815,6 +1823,52 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartWhoseReplacementFitsNowhereIsGivenUp(t *testing.T) {
|
||||
// The profile was edited to need more cards than the node has: stopping
|
||||
// the old replica would leave the model with nothing at all.
|
||||
m := testModel(1)
|
||||
h := newHarness(t, m, gpuNode("1", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
old := h.r.ReadyReplicas()[0].ID
|
||||
|
||||
m.Profiles[0].TP, m.Profiles[0].GPUsPerNode = 8, 8
|
||||
h.setModels(t, m)
|
||||
if err := h.r.Restart(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for range 3 {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.advance(h.r.cfg.DrainGrace + time.Second)
|
||||
}
|
||||
if rep, ok := replicaByID(h, old); !ok || rep.State != Ready {
|
||||
t.Fatalf("the only replica was stopped for a replacement that fits nowhere: %+v", rep)
|
||||
}
|
||||
if st := h.r.RestartState(old); !strings.HasPrefix(st, "refused: ") || !strings.Contains(st, "fits on no node") {
|
||||
t.Errorf("restart state %q should say it was given up, and why", st)
|
||||
}
|
||||
// Asked again, it is tried again.
|
||||
if err := h.r.Restart(old); err != nil || h.r.RestartState(old) != "queued" {
|
||||
t.Errorf("a new restart did not clear the refusal: %q, %v", h.r.RestartState(old), err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartAskedInPlaceStopsFirst(t *testing.T) {
|
||||
// Room beside it, but in place was asked for: an engine that must not
|
||||
// run twice, or memory somebody knows is short.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
old := h.r.ReadyReplicas()[0].ID
|
||||
if err := h.r.RestartInPlace(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
|
||||
t.Fatalf("asked in place, it kept serving beside its replacement: %+v", rep)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
|
||||
h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
|
||||
for i := 0; i < 3; i++ {
|
||||
|
||||
@@ -6,17 +6,34 @@ import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// retire finishes restarting replicas.
|
||||
//
|
||||
// A retiring replica keeps serving until its replacement is ready, and is then
|
||||
// drained like any other. When no replacement could be placed at all —
|
||||
// nowhere else has the memory, which on a small CPU fleet is the usual case —
|
||||
// it is stopped after DrainGrace so its replacement can take its place: a
|
||||
// restart in place, with the model unserved while it loads if it had no other
|
||||
// replica. Said in the log, since that is the one case with an outage.
|
||||
func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
|
||||
// drained like any other. When no replacement could be placed beside it, what
|
||||
// happens depends on what is in the way:
|
||||
//
|
||||
// - the replica itself — one node, or every card that fits held: it is
|
||||
// stopped now, and its replacement takes its place. Waiting would change
|
||||
// nothing; it used to wait DrainGrace anyway, five minutes reading only
|
||||
// "restart replacing", which looked like a restart that never would.
|
||||
// - more than it: the replacement fits nowhere even with it stopped — its
|
||||
// profile was edited to need more than any node has free. Stopping it
|
||||
// would leave the model with nothing, so the restart is given up, and
|
||||
// says why where the replica is shown.
|
||||
// - something being moved out of the way (a preemption, an idle model
|
||||
// stopping): waited for.
|
||||
// - nothing tried this pass (a backoff): the old rule, stopped after
|
||||
// DrainGrace with no replacement in sight.
|
||||
//
|
||||
// A restart asked for in place (ctl restart --in-place) stops first, always.
|
||||
// Each stop is said in the log, since it is the one case with an outage.
|
||||
func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string, readyLive, live, desired int,
|
||||
outcome scaleOutcome, parked bool,
|
||||
) {
|
||||
for _, id := range ids {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -24,6 +41,7 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
|
||||
if ok {
|
||||
since = rep.RetiringAt
|
||||
}
|
||||
inPlace := r.restartInPlace[id]
|
||||
r.mu.Unlock()
|
||||
if !ok {
|
||||
continue
|
||||
@@ -31,15 +49,48 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
|
||||
switch {
|
||||
case readyLive >= desired:
|
||||
r.drainForRestart(ctx, id)
|
||||
case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
|
||||
r.log.Warn("restarting a replica in place: no room for its replacement beside it",
|
||||
case inPlace:
|
||||
r.log.Warn("restarting a replica in place, as asked: stopped before its replacement starts", "replica", id)
|
||||
r.stopForRestart(ctx, id)
|
||||
case live >= desired:
|
||||
// A replacement is placed and loading; it is waited for.
|
||||
case parked:
|
||||
r.refuseRestart(id, "its replacements fail to start; the model is parked (llmgw ctl retry "+m.Name+")")
|
||||
case outcome == noRoom && r.fitsWithout(m, nil, []victim{{ID: id, Model: m.Name}}):
|
||||
r.log.Warn("restarting a replica in place: it is the only thing in its replacement's way",
|
||||
"replica", id, "model", m.Name)
|
||||
r.stopForRestart(ctx, id)
|
||||
case outcome == noRoom || outcome == overLimit:
|
||||
r.refuseRestart(id, "its replacement fits on no node even with this replica stopped "+
|
||||
"(llmgw ctl explain-placement "+m.Name+")")
|
||||
case outcome == notTried && r.now().Sub(since) > r.cfg.DrainGrace:
|
||||
r.log.Warn("restarting a replica in place: no replacement beside it",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
r.drainForRestart(ctx, id)
|
||||
r.stopReplica(ctx, id)
|
||||
r.stopForRestart(ctx, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// stopForRestart drains a replica and stops it at once, for its replacement
|
||||
// to take its place.
|
||||
func (r *Reconciler) stopForRestart(ctx context.Context, id string) {
|
||||
r.drainForRestart(ctx, id)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
|
||||
// refuseRestart gives up a replica's restart, leaving it serving, and keeps
|
||||
// why for whoever looks at the replica.
|
||||
func (r *Reconciler) refuseRestart(id, why string) {
|
||||
r.mu.Lock()
|
||||
if rep, ok := r.replicas[id]; ok {
|
||||
rep.RetiringAt = time.Time{}
|
||||
}
|
||||
delete(r.restartInPlace, id)
|
||||
r.restartRefused[id] = why
|
||||
r.mu.Unlock()
|
||||
r.log.Warn("not restarting a replica: it keeps serving", "replica", id, "because", why)
|
||||
}
|
||||
|
||||
func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -48,6 +99,7 @@ func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
|
||||
rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
|
||||
shards = slices.Clone(rep.Shards)
|
||||
}
|
||||
delete(r.restartInPlace, id)
|
||||
r.mu.Unlock()
|
||||
for _, s := range shards {
|
||||
if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
|
||||
@@ -78,7 +130,14 @@ var ErrNoSuchReplica = errors.New("no ready replica with that id")
|
||||
// Restart queues replicas to be replaced, each by a new one started beside it
|
||||
// before it is stopped — to pick up an edited profile, an engine key, a new
|
||||
// engine version. Replicas of one model are replaced one at a time.
|
||||
func (r *Reconciler) Restart(ids ...string) error {
|
||||
func (r *Reconciler) Restart(ids ...string) error { return r.restart(false, ids) }
|
||||
|
||||
// RestartInPlace queues replicas to be replaced by stopping each first and
|
||||
// then starting its replacement: for when there is knowingly no room beside
|
||||
// it, and the outage while it loads is accepted.
|
||||
func (r *Reconciler) RestartInPlace(ids ...string) error { return r.restart(true, ids) }
|
||||
|
||||
func (r *Reconciler) restart(inPlace bool, ids []string) error {
|
||||
r.mu.Lock()
|
||||
for _, id := range ids {
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -86,6 +145,10 @@ func (r *Reconciler) Restart(ids ...string) error {
|
||||
r.mu.Unlock()
|
||||
return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
|
||||
}
|
||||
delete(r.restartRefused, id)
|
||||
if inPlace {
|
||||
r.restartInPlace[id] = true
|
||||
}
|
||||
if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
|
||||
r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
|
||||
}
|
||||
@@ -95,8 +158,8 @@ func (r *Reconciler) Restart(ids ...string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// RestartState says where a replica's restart stands: "queued", "replacing",
|
||||
// or "" when none was asked for.
|
||||
// RestartState says where a replica's restart stands: "queued",
|
||||
// "replacing", "in place", "refused: why", or "" when none was asked for.
|
||||
func (r *Reconciler) RestartState(id string) string {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
@@ -104,10 +167,14 @@ func (r *Reconciler) RestartState(id string) string {
|
||||
switch {
|
||||
case !ok:
|
||||
return ""
|
||||
case !rep.RetiringAt.IsZero() && r.restartInPlace[id]:
|
||||
return "in place"
|
||||
case !rep.RetiringAt.IsZero():
|
||||
return "replacing"
|
||||
case slices.Contains(r.restartQueue[rep.Model], id):
|
||||
return "queued"
|
||||
case r.restartRefused[id] != "":
|
||||
return "refused: " + r.restartRefused[id]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
+71
-16
@@ -10,6 +10,7 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
@@ -86,10 +87,12 @@ type Facts struct {
|
||||
|
||||
// What installing vLLM here would meet: the version this llmgw installs,
|
||||
// where, the space there, and whether it already has.
|
||||
VLLMVersion string `json:"vllm_version,omitempty"`
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
VLLMVersion string `json:"vllm_version,omitempty"`
|
||||
// DisabledEngines are those agent.yaml says this node never runs.
|
||||
DisabledEngines []string `json:"disabled_engines,omitempty"`
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
}
|
||||
|
||||
// Modules checked, and why each matters.
|
||||
@@ -103,6 +106,10 @@ type Options struct {
|
||||
// VLLMVersion is the vLLM this llmgw installs; its prerequisites are
|
||||
// checked when set.
|
||||
VLLMVersion string
|
||||
// Fabrics are fabrics about to be declared, by name, with their
|
||||
// adapters: what `node add --fabric` will write into agent.yaml, which
|
||||
// does not exist yet when doctor runs before it.
|
||||
Fabrics map[string][]string
|
||||
}
|
||||
|
||||
// Collect reads the node.
|
||||
@@ -183,6 +190,14 @@ func Collect(ctx context.Context, o Options) Facts {
|
||||
}
|
||||
|
||||
f.readConfig(o)
|
||||
if len(o.Fabrics) > 0 {
|
||||
declared := map[string][]string{}
|
||||
for _, c := range f.Fabrics {
|
||||
declared[c.Fabric] = c.Declared
|
||||
}
|
||||
maps.Copy(declared, o.Fabrics)
|
||||
f.Fabrics = discovery.CheckFabrics(declared, f.HCAs)
|
||||
}
|
||||
f.Host.MinComputeCapability = discovery.MinComputeCapability(f.GPUs)
|
||||
if f.VLLMVersion = o.VLLMVersion; f.VLLMVersion != "" {
|
||||
if f.EnginesDir == "" {
|
||||
@@ -238,6 +253,7 @@ func (f *Facts) readConfig(o Options) {
|
||||
_, err = net.InterfaceByName(f.MgmtIface)
|
||||
f.IfaceExists = err == nil
|
||||
f.EnginesDir = cfg.EnginesDir
|
||||
f.DisabledEngines = cfg.Engines.Disabled
|
||||
f.LaunchMode = cfg.Engines.VLLM.LaunchMode
|
||||
if f.LaunchMode == "" {
|
||||
f.LaunchMode = "mp"
|
||||
@@ -419,8 +435,15 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
switch {
|
||||
case p.Active() && p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID < 0:
|
||||
add(Fail, "fabric", where+" is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang",
|
||||
"give the interface an IPv4 address, or check the adapter's RoCE mode")
|
||||
// Fatal only for a port a declared fabric uses. An adapter
|
||||
// nothing is configured to use — a RoCE NIC on a node added
|
||||
// for single-node inference — kept nodes out for a fabric
|
||||
// nobody asked for.
|
||||
msg := where + " is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang"
|
||||
if declared[h.Name] == "" {
|
||||
msg += "; no declared fabric uses it, so nothing will until one does"
|
||||
}
|
||||
add(level, "fabric", msg, "give the interface an IPv4 address, or check the adapter's RoCE mode")
|
||||
case p.Active():
|
||||
on := ""
|
||||
switch {
|
||||
@@ -430,17 +453,24 @@ func Evaluate(f Facts) []Finding {
|
||||
on = ", network " + p.Network
|
||||
}
|
||||
add(OK, "fabric", fmt.Sprintf("%s %s at %d Gb/s (%s)%s", where, p.State, p.RateGbps, p.LinkLayer, on), "")
|
||||
// Questions about which fabric a port is on matter to the
|
||||
// fabrics that use it; for a port none does they are noted,
|
||||
// not warned of.
|
||||
quiet := Warn
|
||||
if declared[h.Name] == "" {
|
||||
quiet = Info
|
||||
}
|
||||
if p.LinkLayer == discovery.LinkInfiniBand && p.SM == nil {
|
||||
why := f.SMErrors[where]
|
||||
fix := ""
|
||||
if strings.Contains(why, "infiniband-diags") {
|
||||
fix = "install infiniband-diags"
|
||||
}
|
||||
add(Warn, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
|
||||
add(quiet, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
|
||||
"; without it, separate fabrics given one name cannot be caught", fix)
|
||||
}
|
||||
if p.LinkLayer == discovery.LinkInfiniBand && p.SMLID == 0 {
|
||||
add(Warn, "fabric", where+" is active but reports no subnet manager LID", "")
|
||||
add(quiet, "fabric", where+" is active but reports no subnet manager LID", "")
|
||||
}
|
||||
case strings.EqualFold(p.PhysState, "LinkUp"):
|
||||
add(level, "fabric", where+" has a link but is "+p.State+": no subnet manager has configured it",
|
||||
@@ -479,7 +509,14 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
if len(f.HCAs) > 0 {
|
||||
if !f.Modules["ib_uverbs"] {
|
||||
add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
|
||||
// RDMA is for replicas spanning nodes; a node that declares no
|
||||
// fabric runs none, and is a node without it.
|
||||
if len(declared) > 0 {
|
||||
add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
|
||||
} else {
|
||||
add(Warn, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA, which a fabric declared "+
|
||||
"later would need; this node runs single-node replicas without it", "modprobe ib_uverbs")
|
||||
}
|
||||
}
|
||||
if len(f.GPUs) > 0 && !f.Modules["nvidia_peermem"] && !f.Modules["nv_peer_mem"] {
|
||||
add(Warn, "fabric", "nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory",
|
||||
@@ -525,7 +562,9 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
}
|
||||
// --- engines ---
|
||||
if f.VLLMVersion != "" {
|
||||
// An engine disabled on this node: whether it could be installed is not
|
||||
// a question anybody is asking.
|
||||
if f.VLLMVersion != "" && !slices.Contains(f.DisabledEngines, "vllm") {
|
||||
vllmFindings(f, add)
|
||||
}
|
||||
|
||||
@@ -542,28 +581,44 @@ func Evaluate(f Facts) []Finding {
|
||||
// the same decision the agent makes, so the report and the node agree.
|
||||
func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
v := f.VLLMVersion
|
||||
b, err := provision.Recipes["vllm"].Choose(v, f.Host)
|
||||
builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
|
||||
var b provision.Build
|
||||
if err == nil {
|
||||
// The first the agent would try; it moves to the next by itself if
|
||||
// that one does not work here.
|
||||
b = builds[0]
|
||||
if b.Version != "" {
|
||||
v = b.Version
|
||||
}
|
||||
}
|
||||
var nh provision.NotHere
|
||||
switch {
|
||||
case errors.As(err, &nh):
|
||||
add(Info, "engines", "vLLM will not be installed: "+nh.Reason, "")
|
||||
return
|
||||
case err != nil:
|
||||
add(Fail, "engines", "vLLM "+v+" cannot be installed: "+err.Error(),
|
||||
"upgrade the NVIDIA driver, or set the control plane's engines.vllm to a version for this one")
|
||||
// A warning, not a failure: the node is still a node, and serves
|
||||
// llama.cpp. Refusing it here kept older GPU generations out of a
|
||||
// cluster for an engine nobody meant to run on them.
|
||||
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
|
||||
"add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
return
|
||||
case f.VLLMInstalled:
|
||||
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
|
||||
return
|
||||
}
|
||||
if f.EnginesFreeGB > 0 && f.EnginesFreeGB < float64(b.FreeGB) {
|
||||
add(Fail, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB",
|
||||
add(Warn, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB; the node joins without it until there is",
|
||||
v, b.FreeGB, f.EnginesDir, f.EnginesFreeGB),
|
||||
"free space there, or point engines_dir in agent.yaml at a larger disk")
|
||||
return
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
|
||||
note := ""
|
||||
if b.Note != "" {
|
||||
note = "; " + b.Note
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
|
||||
}
|
||||
|
||||
func rank(l Level) int {
|
||||
|
||||
@@ -151,8 +151,21 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
says string
|
||||
}{
|
||||
{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
|
||||
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Fail, "driver 580"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Fail, "9.0 GB"},
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
|
||||
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
|
||||
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
// A V100 has no build of vLLM 0.20 or later; it gets the newest
|
||||
// that has Volta kernels, and doctor says which.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
|
||||
{"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
|
||||
VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
|
||||
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
}
|
||||
@@ -200,3 +213,38 @@ func TestDoctorSaysWhenTheCPUHidesAVX2(t *testing.T) {
|
||||
t.Errorf("ARM: %+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineDisabledOnTheNodeIsNotChecked(t *testing.T) {
|
||||
// The field: a V100 node meant for llama.cpp, refused for a vLLM it was
|
||||
// never going to run.
|
||||
f := Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", DisabledEngines: []string{"vllm"}}
|
||||
for _, fd := range Evaluate(f) {
|
||||
if fd.Area == "engines" {
|
||||
t.Errorf("a disabled vLLM was reported on: %s %q", fd.Level, fd.Message)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdapterNoFabricUsesDoesNotKeepTheNodeOut(t *testing.T) {
|
||||
// The field: a RoCE NIC with no v2 GID on a node added for single-node
|
||||
// inference, no fabric given, and the node refused for it.
|
||||
roce := discovery.HCA{Name: "mlx5_2", Ports: []discovery.Port{{Num: 1, State: "ACTIVE",
|
||||
PhysState: "LinkUp", RateGbps: 100, LinkLayer: discovery.LinkEthernet, RoCEv2GID: -1}}}
|
||||
f := healthy()
|
||||
f.HCAs = append(f.HCAs, roce)
|
||||
f.Modules["ib_uverbs"] = false
|
||||
f.Fabrics = nil // nothing declared
|
||||
fs := Evaluate(f)
|
||||
if fails := levels(fs, Fail); len(fails) > 0 {
|
||||
t.Errorf("an undeclared adapter failed the node: %v", fails)
|
||||
}
|
||||
has(t, fs, Warn, "no declared fabric uses it")
|
||||
has(t, fs, Warn, "runs single-node replicas without it")
|
||||
|
||||
// Declared, the same adapter and module are what the fabric needs.
|
||||
f.Fabrics = discovery.CheckFabrics(map[string][]string{"roce-a": {"mlx5_2"}}, f.HCAs)
|
||||
fs = Evaluate(f)
|
||||
has(t, fs, Fail, "mlx5_2/1 is RoCE with no RoCE v2 GID entry")
|
||||
has(t, fs, Fail, "ib_uverbs is not loaded")
|
||||
}
|
||||
@@ -258,6 +258,41 @@ func (f File) IsWeights() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// companionNames are the files beside safetensors weights that an engine
|
||||
// loading the directory reads: the model's shape, its tokenizer, how it was
|
||||
// quantised, and which shard holds which tensor.
|
||||
var companionNames = map[string]bool{
|
||||
"config.json": true, "generation_config.json": true,
|
||||
"tokenizer.json": true, "tokenizer_config.json": true, "tokenizer.model": true,
|
||||
"special_tokens_map.json": true, "added_tokens.json": true,
|
||||
"vocab.json": true, "vocab.txt": true, "merges.txt": true,
|
||||
"chat_template.jinja": true, "chat_template.json": true,
|
||||
"preprocessor_config.json": true, "processor_config.json": true,
|
||||
"quantize_config.json": true, "quant_config.json": true,
|
||||
}
|
||||
|
||||
// Companions are the files of a repository an engine reads beside a
|
||||
// safetensors set: config.json and the tokenizer, the shard index. Only
|
||||
// those at the top: the directory an engine is given is one level deep.
|
||||
//
|
||||
// Never code (a .py): a model that needs its own Python to load asks the
|
||||
// engine to run what the repository ships, which is a decision for whoever
|
||||
// runs the engine, made on purpose, not a file swept in with the tokenizer.
|
||||
func Companions(files []File) []File {
|
||||
var out []File
|
||||
for _, f := range files {
|
||||
name := f.Path
|
||||
if strings.Contains(name, "/") {
|
||||
continue
|
||||
}
|
||||
if companionNames[name] || strings.HasSuffix(name, ".safetensors.index.json") ||
|
||||
strings.HasSuffix(name, ".tiktoken") {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Repo is a repository's current revision and what is in it.
|
||||
type Repo struct {
|
||||
ID string
|
||||
|
||||
@@ -94,6 +94,13 @@ func addWith(ctx context.Context, client *http.Client, root, source string, e En
|
||||
_ = os.Remove(part)
|
||||
return Added{}, err
|
||||
}
|
||||
// The checksum beside the file, before the file is in place: a manifest
|
||||
// listing the file then always has its checksum to hand on, and a node
|
||||
// can tell a file damaged on the way from the one that was mirrored.
|
||||
if err := writeChecksum(final, sum); err != nil {
|
||||
_ = os.Remove(part)
|
||||
return Added{}, err
|
||||
}
|
||||
if err := os.Rename(part, final); err != nil {
|
||||
return Added{}, err
|
||||
}
|
||||
@@ -146,6 +153,9 @@ func Remove(root string, e Entry) error {
|
||||
if err := os.Remove(path); err != nil {
|
||||
return err
|
||||
}
|
||||
// Its checksum goes with it; left, it would keep the directory from
|
||||
// being empty, and so from going.
|
||||
_ = os.Remove(checksumPath(path))
|
||||
versionDir := filepath.Dir(path)
|
||||
if empty, _ := isEmpty(versionDir); empty {
|
||||
_ = os.Remove(versionDir)
|
||||
|
||||
+102
-2
@@ -16,12 +16,15 @@ package mirror
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
@@ -91,7 +94,7 @@ func List(root string) ([]Entry, error) {
|
||||
continue
|
||||
}
|
||||
for _, f := range files {
|
||||
if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) {
|
||||
if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) || strings.HasPrefix(f.Name(), ".") {
|
||||
continue
|
||||
}
|
||||
info, err := f.Info()
|
||||
@@ -169,6 +172,31 @@ func normalizeURL(u string) string {
|
||||
|
||||
const partSuffix = ".part"
|
||||
|
||||
// checksumPath is where a file's sha256 is kept: beside it, hidden, so no
|
||||
// listing takes it for a file of the model and no name a person gives a
|
||||
// file (which may not start with a dot) can be the same.
|
||||
func checksumPath(file string) string {
|
||||
return filepath.Join(filepath.Dir(file), "."+filepath.Base(file)+".sha256")
|
||||
}
|
||||
|
||||
func writeChecksum(file, sum string) error {
|
||||
tmp := checksumPath(file) + partSuffix
|
||||
if err := os.WriteFile(tmp, []byte(sum+"\n"), 0o644); err != nil { //nolint:gosec // a checksum, for anyone to read
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, checksumPath(file))
|
||||
}
|
||||
|
||||
// readChecksum is a file's recorded sha256, or "" for a file put in place by
|
||||
// hand, which has none.
|
||||
func readChecksum(file string) string {
|
||||
b, err := os.ReadFile(checksumPath(file)) //nolint:gosec // beside a file in the mirror
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
// sha256File is the checksum of a file on disk.
|
||||
//
|
||||
// Reported after a download so an operator has something to compare against
|
||||
@@ -205,8 +233,16 @@ type Manifest struct {
|
||||
type ManifestFile struct {
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
// SHA256 is the file's checksum when the mirror fetched it, for a node
|
||||
// to check what it downloaded; empty for a file put there by hand.
|
||||
SHA256 string `json:"sha256,omitempty"`
|
||||
}
|
||||
|
||||
// ErrIncomplete is a version not yet whole: a download still running, or a
|
||||
// shard its own index names missing. Its manifest is not given out, so no
|
||||
// node starts a model from part of it.
|
||||
var ErrIncomplete = errors.New("this version is not complete")
|
||||
|
||||
// ManifestFor lists the files of one model version.
|
||||
//
|
||||
// Built from the directory rather than from a file written alongside it. A
|
||||
@@ -219,15 +255,79 @@ func ManifestFor(root, model, version string) (Manifest, error) {
|
||||
return Manifest{}, err
|
||||
}
|
||||
m := Manifest{Model: model, Version: version}
|
||||
dir := filepath.Join(root, model, version)
|
||||
for _, e := range entries {
|
||||
if e.Model != model || e.Version != version {
|
||||
continue
|
||||
}
|
||||
m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes})
|
||||
m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes,
|
||||
SHA256: readChecksum(filepath.Join(dir, e.File))})
|
||||
m.Bytes += e.Bytes
|
||||
}
|
||||
if len(m.Files) == 0 {
|
||||
return Manifest{}, fmt.Errorf("%s/%s: %w", model, version, os.ErrNotExist)
|
||||
}
|
||||
if err := complete(dir, m.Files); err != nil {
|
||||
return Manifest{}, fmt.Errorf("%s/%s: %w: %v", model, version, ErrIncomplete, err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// complete says whether a version's files are the whole of it: nothing still
|
||||
// downloading, every shard a safetensors index maps a tensor to, and every
|
||||
// part of a split ("-00002-of-00004").
|
||||
//
|
||||
// From what the files say of each other rather than from a list written when
|
||||
// the fetch began, so a version filled by hand is held to the same.
|
||||
func complete(dir string, files []ManifestFile) error {
|
||||
if parts, _ := filepath.Glob(filepath.Join(dir, "*"+partSuffix)); len(parts) > 0 {
|
||||
var names []string
|
||||
for _, p := range parts {
|
||||
if b := filepath.Base(p); !strings.HasPrefix(b, ".") {
|
||||
names = append(names, strings.TrimSuffix(b, partSuffix))
|
||||
}
|
||||
}
|
||||
if len(names) > 0 {
|
||||
return fmt.Errorf("still downloading %s", strings.Join(names, ", "))
|
||||
}
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, f := range files {
|
||||
have[f.Name] = true
|
||||
}
|
||||
var missing []string
|
||||
want := map[string]bool{}
|
||||
for _, f := range files {
|
||||
if strings.HasSuffix(f.Name, ".safetensors.index.json") {
|
||||
var idx struct {
|
||||
WeightMap map[string]string `json:"weight_map"`
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, f.Name)) //nolint:gosec // a file in the mirror
|
||||
if err != nil || json.Unmarshal(b, &idx) != nil {
|
||||
return fmt.Errorf("%s cannot be read", f.Name)
|
||||
}
|
||||
for _, shard := range idx.WeightMap {
|
||||
want[shard] = true
|
||||
}
|
||||
}
|
||||
if m := splitPart.FindStringSubmatch(f.Name); m != nil {
|
||||
total, _ := strconv.Atoi(m[3])
|
||||
for i := 1; i <= total; i++ {
|
||||
want[fmt.Sprintf("%s-%0*d-of-%s%s", m[1], len(m[2]), i, m[3], m[4])] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
for name := range want {
|
||||
if !have[name] {
|
||||
missing = append(missing, name)
|
||||
}
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
sort.Strings(missing)
|
||||
return fmt.Errorf("missing %s", strings.Join(missing, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// splitPart matches one part of a split: prefix, part, total, extension.
|
||||
var splitPart = regexp.MustCompile(`^(.+)-(\d{5})-of-(\d{5})(\.[A-Za-z0-9]+)$`)
|
||||
@@ -177,3 +177,63 @@ func TestANameThatLeavesTheMirrorIsRefused(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAVersionIsGivenOutOnlyWhole(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "m", "v1")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write := func(name, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
write("config.json", "{}")
|
||||
write("model.safetensors.index.json",
|
||||
`{"weight_map":{"a":"model-00001-of-00002.safetensors","b":"model-00002-of-00002.safetensors"}}`)
|
||||
write("model-00001-of-00002.safetensors", "aa")
|
||||
write("model-00002-of-00002.safetensors.part", "b")
|
||||
|
||||
if _, err := ManifestFor(root, "m", "v1"); !errors.Is(err, ErrIncomplete) || !strings.Contains(err.Error(), "still downloading") {
|
||||
t.Fatalf("a version with a shard downloading was given out: %v", err)
|
||||
}
|
||||
// The download failed and its .part went: the index still names it.
|
||||
_ = os.Remove(filepath.Join(dir, "model-00002-of-00002.safetensors.part"))
|
||||
if _, err := ManifestFor(root, "m", "v1"); !errors.Is(err, ErrIncomplete) ||
|
||||
!strings.Contains(err.Error(), "missing model-00002-of-00002.safetensors") {
|
||||
t.Fatalf("a version missing a shard its index names was given out: %v", err)
|
||||
}
|
||||
write("model-00002-of-00002.safetensors", "bb")
|
||||
m, err := ManifestFor(root, "m", "v1")
|
||||
if err != nil || len(m.Files) != 4 {
|
||||
t.Fatalf("the whole version: %+v, %v", m, err)
|
||||
}
|
||||
|
||||
// A split GGUF with no index is held to its own numbering.
|
||||
gg := filepath.Join(root, "g", "v1")
|
||||
_ = os.MkdirAll(gg, 0o755)
|
||||
_ = os.WriteFile(filepath.Join(gg, "q-00001-of-00003.gguf"), []byte("x"), 0o644)
|
||||
_ = os.WriteFile(filepath.Join(gg, "q-00003-of-00003.gguf"), []byte("x"), 0o644)
|
||||
if _, err := ManifestFor(root, "g", "v1"); !errors.Is(err, ErrIncomplete) || !strings.Contains(err.Error(), "q-00002-of-00003.gguf") {
|
||||
t.Fatalf("a split missing its second part was given out: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMirroredFilesChecksumIsHandedOnAndHiddenFromListings(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
e := Entry{Model: "m", Version: "v1", File: "f.gguf"}
|
||||
added, err := Add(context.Background(), root, serving(t, "weights"), e, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := List(root)
|
||||
if len(entries) != 1 {
|
||||
t.Errorf("the checksum was listed as a file of the model: %+v", entries)
|
||||
}
|
||||
m, err := ManifestFor(root, "m", "v1")
|
||||
if err != nil || m.Files[0].SHA256 != added.SHA256 || added.SHA256 == "" {
|
||||
t.Errorf("manifest %+v, checksum %q: %v", m, added.SHA256, err)
|
||||
}
|
||||
}
|
||||
@@ -18,7 +18,6 @@ import (
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -27,6 +26,8 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
@@ -86,6 +87,15 @@ type Options struct {
|
||||
Releases string
|
||||
// IgnoreChecks goes on when doctor finds a problem.
|
||||
IgnoreChecks bool
|
||||
// DisableEngines are engines this node never installs or advertises,
|
||||
// written into its agent.yaml, and not checked by doctor: vLLM on a node
|
||||
// of cards it does not run on, meant for llama.cpp alone. Narrower than
|
||||
// IgnoreChecks, which also passes over everything else.
|
||||
DisableEngines []string
|
||||
// VLLMVersion is the vLLM the control plane has nodes install, for
|
||||
// doctor to check against; "none" when it has them install none, empty
|
||||
// for doctor's own default.
|
||||
VLLMVersion string
|
||||
// SSHWait is how long an unreachable machine is waited for.
|
||||
SSHWait time.Duration
|
||||
|
||||
@@ -390,6 +400,18 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
|
||||
if r.Opts.IBRoot != "" {
|
||||
args += " --ib-root " + shellQuote(r.Opts.IBRoot)
|
||||
}
|
||||
// The fabrics given, checked as the ones they will be; any adapter they
|
||||
// do not name is checked as one nothing uses.
|
||||
for _, fab := range r.Opts.Fabrics {
|
||||
args += " --fabric " + shellQuote(fab.Name+":"+strings.Join(fab.HCAs, ","))
|
||||
}
|
||||
switch {
|
||||
case slices.Contains(r.Opts.DisableEngines, "vllm"):
|
||||
args += " --vllm none"
|
||||
r.emit(StepCheck, Info, "vLLM is disabled on this node: it joins with llama.cpp alone", "")
|
||||
case r.Opts.VLLMVersion != "":
|
||||
args += " --vllm " + shellQuote(r.Opts.VLLMVersion)
|
||||
}
|
||||
out, err := r.Remote.Run(ctx, args, nil)
|
||||
var rep struct {
|
||||
Findings []doctor.Finding `json:"findings"`
|
||||
@@ -596,8 +618,11 @@ func AgentYAML(o Options, res *Result) string {
|
||||
fmt.Fprintf(&b, " ib_root: %q\n", o.IBRoot)
|
||||
}
|
||||
}
|
||||
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" {
|
||||
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" || len(o.DisableEngines) > 0 {
|
||||
b.WriteString("engines:\n")
|
||||
if len(o.DisableEngines) > 0 {
|
||||
fmt.Fprintf(&b, " disabled: [%s]\n", strings.Join(o.DisableEngines, ", "))
|
||||
}
|
||||
if o.LlamaCppBin != "" {
|
||||
fmt.Fprintf(&b, " llamacpp:\n bin: %q\n", o.LlamaCppBin)
|
||||
}
|
||||
|
||||
@@ -280,3 +280,44 @@ func said(events []Event, level, s string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func TestAnEngineDisabledOnANodeIsWrittenAndNotChecked(t *testing.T) {
|
||||
m := newFake()
|
||||
r, events := runner(t, m, Options{Version: "dev", DisableEngines: []string{"vllm"}, VLLMVersion: "0.30.0"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if yaml := string(m.files["/etc/llmgw/agent.yaml"]); !strings.Contains(yaml, "engines:\n disabled: [vllm]") {
|
||||
t.Errorf("agent.yaml:\n%s", yaml)
|
||||
}
|
||||
if !m.ran("sudo llmgw doctor --json --vllm none") {
|
||||
t.Errorf("doctor was asked about a vLLM this node will never run: %v", m.scripts)
|
||||
}
|
||||
if !said(*events, Info, "joins with llama.cpp alone") {
|
||||
t.Error("the reduced engine set went unsaid")
|
||||
}
|
||||
|
||||
// Not disabled: doctor checks the vLLM the control plane installs, not
|
||||
// its own default.
|
||||
m = newFake()
|
||||
r, _ = runner(t, m, Options{Version: "dev", VLLMVersion: "0.25.1"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !m.ran("sudo llmgw doctor --json --vllm '0.25.1'") && !m.ran("sudo llmgw doctor --json --vllm 0.25.1") {
|
||||
t.Errorf("doctor was not told the control plane's vLLM: %v", m.scripts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheFabricsGivenAreCheckedAsDeclared(t *testing.T) {
|
||||
// agent.yaml is written after doctor runs, so the fabrics on the command
|
||||
// line are handed to doctor: checked strictly, and nothing else is.
|
||||
m := newFake()
|
||||
r, _ := runner(t, m, Options{Version: "dev", Fabrics: []Fabric{{Name: "ib-a", HCAs: []string{"mlx5_0", "mlx5_1"}}}})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !m.ran("--fabric 'ib-a:mlx5_0,mlx5_1'") {
|
||||
t.Errorf("doctor was not told the fabric: %v", m.scripts)
|
||||
}
|
||||
}
|
||||
+13
-1
@@ -255,6 +255,18 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
}
|
||||
seen := map[string]struct{}{}
|
||||
var out []web.ChatModel
|
||||
// What each model's replicas compute on, across all of them: one model
|
||||
// can run on a card and on CPUs at once.
|
||||
hardware := map[string][]string{}
|
||||
for _, s := range r.reg.Snapshots("") {
|
||||
hw := hardwareOf(s.Pool)
|
||||
for _, m := range s.Models {
|
||||
name, _ := r.resolve(m)
|
||||
if hw != "" && !slices.Contains(hardware[name], hw) {
|
||||
hardware[name] = append(hardware[name], hw)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, s := range r.reg.Snapshots("") {
|
||||
for _, m := range s.Models {
|
||||
// Resolved rather than taken as sent: an alias and the name it
|
||||
@@ -265,7 +277,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated()})
|
||||
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -196,8 +196,16 @@ func (r *Router) Handler() http.Handler {
|
||||
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
|
||||
pages.SetCapacity(c)
|
||||
}
|
||||
// The engine build each GPU node installs, chosen on the cluster
|
||||
// page.
|
||||
if e, ok := r.clusterView.(web.NodeEngineAdmin); ok {
|
||||
pages.RegisterNodeEngines(mux, e)
|
||||
}
|
||||
pages.RegisterAdminQuotas(mux, r.store, r.acct)
|
||||
// And the report of what that tree actually used.
|
||||
if r.history != nil {
|
||||
pages.SetHistory(r.history)
|
||||
}
|
||||
pages.RegisterAdminUsage(mux, r.store, r.acct)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
|
||||
func TestAReplySaysWhatHardwareAnsweredIt(t *testing.T) {
|
||||
// qwen2.5-coder-14b had one replica on cuda01's card and one on a CPU
|
||||
// node, and the chat page could not tell which had answered: the same
|
||||
// model, ten times faster or slower, for no visible reason.
|
||||
for pool, want := range map[string]string{
|
||||
"none/6000-24g": "6000-24g",
|
||||
"none/cpu": "cpu",
|
||||
"": "",
|
||||
} {
|
||||
e := newFakeEngine(t, "m")
|
||||
rt := newTestRouter(t, baseConfig())
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "r", URL: e.srv.URL, Models: []string{"m"}, Pool: pool, Weight: 1},
|
||||
})
|
||||
waitForEndpoint(t, rt)
|
||||
|
||||
rec := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("%q: status %d: %s", pool, rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := rec.Header().Get(hardwareHeader); got != want {
|
||||
t.Errorf("pool %q: hardware %q, want %q", pool, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestThePickerSaysWhereEachModelRuns(t *testing.T) {
|
||||
gpu, cpu := newFakeEngine(t, "qwen2.5-coder-7b-instruct"), newFakeEngine(t, "qwen2.5-coder-7b-instruct")
|
||||
rt, user := chatRouter(t, gpu.srv.URL)
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "g", URL: gpu.srv.URL, Models: []string{"qwen2.5-coder-7b-instruct"}, Pool: "none/6000-24g", Weight: 1},
|
||||
{ID: "c", URL: cpu.srv.URL, Models: []string{"coder"}, Pool: "none/cpu", Weight: 1},
|
||||
})
|
||||
waitForEndpoint(t, rt)
|
||||
|
||||
got := rt.modelsFor(httptest.NewRequest("GET", "/chat", nil), user)
|
||||
if len(got) != 1 || got[0].Runs() != "GPU + CPU" {
|
||||
t.Fatalf("picker %+v; want one model running on GPU + CPU, its alias's replica included", got)
|
||||
}
|
||||
}
|
||||
@@ -749,6 +749,16 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
if a, ok := r.adapter(req.Model); ok && a.base == model {
|
||||
body = withAdapter(body, model, a.index, a.scale)
|
||||
}
|
||||
// What computes the answer, said before the proxy writes the headers.
|
||||
// A model can have replicas on cards and on CPUs at once, and the two
|
||||
// differ tenfold in speed: a reply that came slowly is otherwise
|
||||
// indistinguishable from a model that is slow everywhere. Set again on
|
||||
// a retry, which only happens before anything was written.
|
||||
if hw := hardwareOf(dec.Endpoint.Pool); hw != "" {
|
||||
w.Header().Set(hardwareHeader, hw)
|
||||
} else {
|
||||
w.Header().Del(hardwareHeader)
|
||||
}
|
||||
out, perr := r.px.Do(uctx, w, proxy.Request{
|
||||
URL: dec.Endpoint.URL + req.Path,
|
||||
APIKey: dec.Endpoint.APIKey,
|
||||
@@ -791,6 +801,21 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
return last
|
||||
}
|
||||
|
||||
// hardwareHeader says what the replica that answered computes on: "cpu", or
|
||||
// the GPU class, such as "6000-24g".
|
||||
const hardwareHeader = "x-llmgw-hardware"
|
||||
|
||||
// hardwareOf reads it from a pool, which placement names fabric/class —
|
||||
// "none/cpu", "none/6000-24g", "ib-a/h100-80g". Empty for an endpoint
|
||||
// configured by hand, which has no pool and says nothing about its hardware.
|
||||
func hardwareOf(pool string) string {
|
||||
i := strings.LastIndexByte(pool, '/')
|
||||
if i < 0 || i == len(pool)-1 {
|
||||
return ""
|
||||
}
|
||||
return pool[i+1:]
|
||||
}
|
||||
|
||||
func without(eps []endpoint.Snapshot, id string) []endpoint.Snapshot {
|
||||
return slices.DeleteFunc(slices.Clone(eps), func(e endpoint.Snapshot) bool { return e.ID == id })
|
||||
}
|
||||
|
||||
@@ -45,6 +45,7 @@ var copyOrder = []string{
|
||||
// derived from this, and a new secret would lock the router out of all
|
||||
// of them at the moment of the move.
|
||||
"cluster_secrets",
|
||||
"node_engines",
|
||||
}
|
||||
|
||||
// notCopied are tables deliberately left behind.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
-- The build of an engine chosen for a node, where it is not left to the
|
||||
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
|
||||
-- or no vLLM at all. A node with no row here installs what its hardware runs.
|
||||
CREATE TABLE node_engines (
|
||||
node_id TEXT NOT NULL,
|
||||
engine TEXT NOT NULL,
|
||||
variant TEXT NOT NULL,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (node_id, engine)
|
||||
);
|
||||
@@ -0,0 +1,11 @@
|
||||
-- The build of an engine chosen for a node, where it is not left to the
|
||||
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
|
||||
-- or no vLLM at all. A node with no row here installs what its hardware runs.
|
||||
CREATE TABLE node_engines (
|
||||
node_id TEXT NOT NULL,
|
||||
engine TEXT NOT NULL,
|
||||
variant TEXT NOT NULL,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (node_id, engine)
|
||||
);
|
||||
@@ -0,0 +1,44 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NodeEngines are the engine builds chosen per node: node id, then engine,
|
||||
// then the build ("cu129") or "none". A node or engine not here is left to
|
||||
// the node's hardware.
|
||||
func (s *DB) NodeEngines(ctx context.Context) (map[string]map[string]string, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `SELECT node_id, engine, variant FROM node_engines`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
out := map[string]map[string]string{}
|
||||
for rows.Next() {
|
||||
var node, engine, variant string
|
||||
if err := rows.Scan(&node, &engine, &variant); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if out[node] == nil {
|
||||
out[node] = map[string]string{}
|
||||
}
|
||||
out[node][engine] = variant
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SetNodeEngine chooses an engine's build for a node; an empty variant
|
||||
// leaves it to the node's hardware again.
|
||||
func (s *DB) SetNodeEngine(ctx context.Context, node, engine, variant, by string) error {
|
||||
if variant == "" {
|
||||
_, err := s.db.ExecContext(ctx, `DELETE FROM node_engines WHERE node_id = ? AND engine = ?`, node, engine)
|
||||
return err
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO node_engines (node_id, engine, variant, updated_at, updated_by) VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT (node_id, engine) DO UPDATE SET variant = excluded.variant,
|
||||
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
|
||||
node, engine, variant, time.Now().UnixMilli(), by)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestAnEngineBuildIsChosenPerNodeAndCanBeLeftToItAgain(t *testing.T) {
|
||||
db := capacityStore(t)
|
||||
ctx := t.Context()
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "cu129", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "none", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := db.NodeEngines(ctx)
|
||||
if err != nil || got["gpu07"]["vllm"] != "none" {
|
||||
t.Fatalf("got %v, %v; the second choice replaces the first", got, err)
|
||||
}
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, _ := db.NodeEngines(ctx); len(got["gpu07"]) != 0 {
|
||||
t.Errorf("left to the node again, still %v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
func TestADuplicateIsRecognisedOnEitherDatabase(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
err error
|
||||
want bool
|
||||
}{
|
||||
{"sqlite", errors.New("constraint failed: UNIQUE constraint failed: users.email (2067)"), true},
|
||||
// What pgx returns, wrapped as database/sql passes it on: "unique"
|
||||
// in lower case, which a check for "UNIQUE" missed.
|
||||
{"postgres", fmt.Errorf("exec: %w", &pgconn.PgError{Code: "23505",
|
||||
Message: `duplicate key value violates unique constraint "users_email_key"`}), true},
|
||||
{"postgres, as text", errors.New(`ERROR: duplicate key value violates unique constraint "users_email_key" (SQLSTATE 23505)`), true},
|
||||
{"another postgres error", &pgconn.PgError{Code: "23502", Message: "null value in column"}, false},
|
||||
{"nothing", nil, false},
|
||||
} {
|
||||
if got := isUniqueViolation(c.err); got != c.want {
|
||||
t.Errorf("%s: %v, want %v", c.name, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
+20
-1
@@ -6,6 +6,8 @@ import (
|
||||
"errors"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
// UserState is where an account is in its life.
|
||||
@@ -66,12 +68,29 @@ func (s *DB) CreateUser(ctx context.Context, u User) error {
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
u.ID, u.Email, u.Name, u.PasswordHash, string(u.State),
|
||||
nullString(u.ProjectID), millis(u.CreatedAt))
|
||||
if err != nil && strings.Contains(err.Error(), "UNIQUE") {
|
||||
if isUniqueViolation(err) {
|
||||
return ErrEmailTaken
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// isUniqueViolation reports whether err is a row refused for a value another
|
||||
// row already has. Each database says it its own way: SQLite in words
|
||||
// ("UNIQUE constraint failed"), PostgreSQL by code, 23505, in a message with
|
||||
// "unique" in lower case — which a check for the upper-case word missed, so a
|
||||
// second sign-up with an address on PostgreSQL was "something went wrong".
|
||||
func isUniqueViolation(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
var pe *pgconn.PgError
|
||||
if errors.As(err, &pe) {
|
||||
return pe.Code == "23505"
|
||||
}
|
||||
m := err.Error()
|
||||
return strings.Contains(m, "UNIQUE constraint failed") || strings.Contains(m, "SQLSTATE 23505")
|
||||
}
|
||||
|
||||
// UserByEmail finds an account by its address.
|
||||
func (s *DB) UserByEmail(ctx context.Context, email string) (User, error) {
|
||||
return s.scanUser(s.db.QueryRowContext(ctx,
|
||||
|
||||
+28
-2
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base32"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
@@ -63,6 +64,9 @@ type AccountPages struct {
|
||||
// hasQuotas is set when /admin/quotas is registered.
|
||||
hasQuotas bool
|
||||
// capacity sets capacity quotas, from the quotas page; nil leaves them off.
|
||||
// history is the metrics store, for what only it knows: the GPU time
|
||||
// each owner held. Nil without one.
|
||||
history History
|
||||
capacity CapacityAdmin
|
||||
// personalOff refuses personal keys; see SetPersonalOff.
|
||||
personalOff bool
|
||||
@@ -84,6 +88,13 @@ type AccountPages struct {
|
||||
// could run it. Both nil where the mirror page is read-only.
|
||||
catalog CatalogAdmin
|
||||
fleet Source
|
||||
// The stores behind the people page's three tabs — the tree and its
|
||||
// people, the limits, the usage — each set when its part is mounted, so
|
||||
// the page draws the parts that exist.
|
||||
peopleStore AccountAdmin
|
||||
quotaStore QuotaAdmin
|
||||
usageStore UsageAdmin
|
||||
usagePriced Priced
|
||||
}
|
||||
|
||||
// ServedModels supplies the names the profile page's example should use.
|
||||
@@ -169,8 +180,12 @@ func (p *AccountPages) signup(w http.ResponseWriter, r *http.Request) {
|
||||
case errors.Is(err, account.ErrWeakPassword), errors.Is(err, account.ErrInvalidEmail):
|
||||
d.Error = err.Error()
|
||||
default:
|
||||
p.log.Error("signup failed", "err", err)
|
||||
d.Error = "something went wrong; try again"
|
||||
// Nothing of the database's words on the page; a reference that finds
|
||||
// them in the log for whoever runs the gateway.
|
||||
ref := errorRef()
|
||||
p.log.Error("signup failed", "ref", ref, "err", err)
|
||||
d.Error = "something went wrong on our side, and your account was not created. " +
|
||||
"Try again later; if it keeps happening, give an operator this reference: " + ref
|
||||
}
|
||||
render(w, "signup", d)
|
||||
}
|
||||
@@ -638,3 +653,14 @@ func (p *AccountPages) renderWithCSRF(w http.ResponseWriter, r *http.Request,
|
||||
|
||||
// SetLlamaCppVersion tells the pages which llama.cpp build the nodes run.
|
||||
func (p *AccountPages) SetLlamaCppVersion(v string) { p.llamaVersion = v }
|
||||
|
||||
// errorRef is a short reference for an error a page does not explain, logged
|
||||
// beside the error itself.
|
||||
func errorRef() string {
|
||||
var b [4]byte
|
||||
_, _ = rand.Read(b[:])
|
||||
return "err-" + hex.EncodeToString(b[:])
|
||||
}
|
||||
|
||||
// SetHistory gives the pages the metrics store.
|
||||
func (p *AccountPages) SetHistory(h History) { p.history = h }
|
||||
@@ -60,6 +60,9 @@ type repoSet struct {
|
||||
// A shard set goes as one action — a fifteen-shard model is not fifteen
|
||||
// decisions, it is one.
|
||||
Sources []repoSource
|
||||
// Companions are the files fetched with a safetensors set that are not
|
||||
// weights: config.json, the tokenizer, the shard index.
|
||||
Companions []string
|
||||
// Model and Version are what the fetch form starts with. Editable, because
|
||||
// the name a catalog entry uses is the operator's to choose and the
|
||||
// registry's idea of it is only a good guess.
|
||||
@@ -317,6 +320,16 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// The CPU while GPUs sit free is a choice to make on purpose:
|
||||
// made by default, it was a model running at a tenth of the
|
||||
// speed with nothing saying why.
|
||||
if model.Profiles[0].CPUOnly() && r.FormValue("cpu_ok") == "" {
|
||||
if n := freeGPUs(p.fleet); n > 0 {
|
||||
return "", fmt.Errorf("this fleet has %s free and %s was set to run on the CPU; "+
|
||||
"choose a GPU placement, or tick \"run on the CPU anyway\"",
|
||||
plural(n, "GPU", "GPUs"), model.Name)
|
||||
}
|
||||
}
|
||||
// The same check `llmgw ctl model add` makes. This page builds
|
||||
// the source out of a row the mirror reported, so it is right
|
||||
// far more often than something typed — but a file removed
|
||||
@@ -332,8 +345,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
// Placed on the next pass, not now: saying it is serving would
|
||||
// be wrong for however long the weights take to reach a node.
|
||||
return fmt.Sprintf(
|
||||
"%s added to the catalog, %s; it is placed on the next pass",
|
||||
model.Name, plural(model.Scaling.Min, "replica", "replicas")), nil
|
||||
"%s added to the catalog, %s: %s; it is placed on the next pass",
|
||||
model.Name, plural(model.Scaling.Min, "replica", "replicas"), describeServe(model)), nil
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -768,6 +781,17 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
|
||||
rs.Sources = append(rs.Sources,
|
||||
repoSource{URL: got.DownloadURL("", f.Path), Path: f.Path})
|
||||
}
|
||||
// Safetensors are not a model on their own: vLLM loads a directory,
|
||||
// and reads its config and tokenizer before any weight. Fetched as
|
||||
// part of the set, or the set arrives as shards nothing can load.
|
||||
if strings.HasSuffix(strings.ToLower(set.Name), ".safetensors") {
|
||||
for _, f := range hf.Companions(got.Files) {
|
||||
rs.Sources = append(rs.Sources,
|
||||
repoSource{URL: got.DownloadURL("", f.Path), Path: f.Path})
|
||||
rs.Companions = append(rs.Companions, f.Path)
|
||||
rs.Bytes += f.Bytes
|
||||
}
|
||||
}
|
||||
d.RepoSets = append(d.RepoSets, rs)
|
||||
}
|
||||
p.probeKinds(r.Context(), d.RepoSets)
|
||||
|
||||
@@ -710,3 +710,37 @@ func TestARefusedFetchReturnsToItsFormAndAStartedOneToTheList(t *testing.T) {
|
||||
t.Errorf("a started download should be shown where it is followed: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASafetensorsSetIsFetchedWithWhatTheEngineReadsBesideIt(t *testing.T) {
|
||||
// The field: a four-shard AWQ model mirrored as its shards alone, and
|
||||
// vLLM failing at start for want of config.json.
|
||||
reg := &fakeRegistry{repo: hf.Repo{
|
||||
ID: "Qwen/Qwen2.5-32B-Instruct-AWQ", Revision: "abc123def456",
|
||||
Files: []hf.File{
|
||||
{Path: "model-00001-of-00002.safetensors", Bytes: 4_000_000_000},
|
||||
{Path: "model-00002-of-00002.safetensors", Bytes: 3_000_000_000},
|
||||
{Path: "model.safetensors.index.json", Bytes: 90_000},
|
||||
{Path: "config.json", Bytes: 900},
|
||||
{Path: "tokenizer.json", Bytes: 7_000_000},
|
||||
{Path: "tokenizer_config.json", Bytes: 7_000},
|
||||
{Path: "README.md", Bytes: 9_000},
|
||||
{Path: "modeling_custom.py", Bytes: 40_000},
|
||||
{Path: "onnx/config.json", Bytes: 900},
|
||||
},
|
||||
}}
|
||||
body := mirrorGet(t, mirrorPageWith(t, &fakeMirror{}, reg, true),
|
||||
"/admin/mirror?repo=Qwen/Qwen2.5-32B-Instruct-AWQ").Body.String()
|
||||
for _, want := range []string{"config.json", "tokenizer.json", "tokenizer_config.json", "model.safetensors.index.json"} {
|
||||
if !strings.Contains(body, `name="file" value="`+want+`"`) {
|
||||
t.Errorf("%s is not fetched with the set", want)
|
||||
}
|
||||
}
|
||||
for _, not := range []string{"README.md", "modeling_custom.py", "onnx/config.json"} {
|
||||
if strings.Contains(body, `name="file" value="`+not+`"`) {
|
||||
t.Errorf("%s was swept in with the set", not)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(body, "with 4 files the engine reads beside them") {
|
||||
t.Errorf("the companions are not said; got %s", body)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// NodeEngineAdmin chooses an engine's build for a node: "auto", a build
|
||||
// such as "cu129", or "none".
|
||||
type NodeEngineAdmin interface {
|
||||
SetNodeEngine(ctx context.Context, node, engine, build, by string) error
|
||||
}
|
||||
|
||||
// RegisterNodeEngines mounts the choice of engine builds on the cluster
|
||||
// page: for an administrator, with the session's CSRF token, and answered by
|
||||
// going back to the page with what happened.
|
||||
func (p *AccountPages) RegisterNodeEngines(mux *http.ServeMux, a NodeEngineAdmin) {
|
||||
mux.HandleFunc("POST /admin/nodes/{id}/engines", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
sess, _, _ := p.session(r)
|
||||
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
node, engine, build := r.PathValue("id"), r.FormValue("engine"), r.FormValue("build")
|
||||
q := url.Values{}
|
||||
if err := a.SetNodeEngine(r.Context(), node, engine, build, admin.Email); err != nil {
|
||||
p.log.Warn("an engine build was not chosen", "admin", admin.Email, "node", node, "err", err)
|
||||
q.Set("problem", node+": "+err.Error())
|
||||
} else {
|
||||
p.log.Info("an engine build was chosen", "admin", admin.Email, "node", node, "engine", engine, "build", build)
|
||||
q.Set("notice", node+" installs "+engine+" "+buildWords(build)+
|
||||
"; the node is told now, and installs it in the background")
|
||||
}
|
||||
http.Redirect(w, r, "/admin?"+q.Encode(), http.StatusSeeOther)
|
||||
}))
|
||||
}
|
||||
|
||||
func buildWords(b string) string {
|
||||
switch b {
|
||||
case "auto":
|
||||
return "in the build its hardware runs"
|
||||
case "none":
|
||||
return "no longer"
|
||||
}
|
||||
return "built for " + map[string]string{"cu13": "CUDA 13", "cu129": "CUDA 12.9"}[b]
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// buildFleet is one GPU node whose vLLM build was chosen, and one CPU node.
|
||||
type buildFleet struct{ fakeFleet }
|
||||
|
||||
func (buildFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{
|
||||
{ID: "gpu07", State: "ready", GPUTotal: 8, GPUFree: 8,
|
||||
EngineBuilds: map[string]string{"vllm": "cu129"},
|
||||
Provision: []inventory.EngineInstall{{Engine: "vllm", Version: "0.30.0", Variant: "cu129", State: "installing"}}},
|
||||
{ID: "cpu01", State: "ready"},
|
||||
}
|
||||
}
|
||||
|
||||
type recordedChoice struct{ node, engine, build, by string }
|
||||
|
||||
type fakeEngineAdmin struct{ got []recordedChoice }
|
||||
|
||||
func (f *fakeEngineAdmin) SetNodeEngine(_ context.Context, node, engine, build, by string) error {
|
||||
f.got = append(f.got, recordedChoice{node, engine, build, by})
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /admin", p.AdminPage(Dashboard(buildFleet{}, nil, "v1", time.Second, false, NavExtras{}, nil)))
|
||||
fe := &fakeEngineAdmin{}
|
||||
p.RegisterNodeEngines(mux, fe)
|
||||
|
||||
body := mirrorGet(t, mux, "/admin").Body.String()
|
||||
if !strings.Contains(body, `action="/admin/nodes/gpu07/engines"`) ||
|
||||
!strings.Contains(body, `<option value="cu129" selected>CUDA 12.9</option>`) {
|
||||
t.Fatalf("no choice of build for the GPU node, with its current one selected: %s", body)
|
||||
}
|
||||
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
|
||||
t.Error("a node with no GPU was offered a vLLM build")
|
||||
}
|
||||
if !strings.Contains(body, "0.30.0 CUDA 12.9 installing") {
|
||||
t.Error("what is installed is not shown beside the choice")
|
||||
}
|
||||
|
||||
// The read-only page on the admin listener has no session, so no form.
|
||||
_, ro := get(t, Dashboard(buildFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(ro, "/engines\"") {
|
||||
t.Error("the admin listener's page offers a change it cannot sign")
|
||||
}
|
||||
|
||||
w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"tok"}, "engine": {"vllm"}, "build": {"none"}})
|
||||
if w.Code != http.StatusSeeOther || len(fe.got) != 1 || fe.got[0] != (recordedChoice{"gpu07", "vllm", "none", "a@example.com"}) {
|
||||
t.Fatalf("status %d, chose %+v", w.Code, fe.got)
|
||||
}
|
||||
if loc := w.Header().Get("Location"); !strings.Contains(loc, "notice=") {
|
||||
t.Errorf("redirected to %s, saying nothing", loc)
|
||||
}
|
||||
if w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"bad"}, "engine": {"vllm"}, "build": {"none"}}); w.Code != http.StatusBadRequest || len(fe.got) != 1 {
|
||||
t.Errorf("a forged form was accepted: %d", w.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,259 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// The people page's tabs. One page, because people, the limits they are held
|
||||
// to and what they used are three views of one tree: managing a team meant its
|
||||
// members on one page, its quota on another and its spend on a third, each
|
||||
// linking to the others. Each tab keeps the address it had as a page of its
|
||||
// own, so links and bookmarks to /admin/quotas and /admin/usage still land.
|
||||
const (
|
||||
tabPeople = "people"
|
||||
tabLimits = "limits"
|
||||
tabUsage = "usage"
|
||||
)
|
||||
|
||||
// peopleView is what a request asks the people page for.
|
||||
type peopleView struct {
|
||||
Tab string
|
||||
Msg, Err string
|
||||
// Open is the project shown expanded in the tree.
|
||||
Open string
|
||||
}
|
||||
|
||||
type peopleData struct {
|
||||
Nav Nav
|
||||
Tab string
|
||||
// The tabs this server has, each built only when it is the one shown.
|
||||
HasPeople, HasLimits, HasUsage bool
|
||||
Users *adminUsersData
|
||||
Quotas *adminQuotasData
|
||||
Usage *adminUsageData
|
||||
// Detail is the branch or person chosen in the tree, nil when none is.
|
||||
Detail *peopleDetail
|
||||
// Sel is what was chosen, "team:<id>" or "user:<id>", for links to keep.
|
||||
Sel string
|
||||
Pending int
|
||||
Version string
|
||||
Now string
|
||||
}
|
||||
|
||||
// peopleDetail is one organization, team, project or person, with everything
|
||||
// the three tabs say about it in one place.
|
||||
type peopleDetail struct {
|
||||
Kind store.Scope
|
||||
ID string
|
||||
// Name is what it is called, Path where it sits.
|
||||
Name, Path string
|
||||
|
||||
// Limits is its own quota row, with what it is held to from above; nil
|
||||
// for a person, or where limits are not mounted.
|
||||
Limits *quotaRow
|
||||
// Person is a person's own limit.
|
||||
Person *personRow
|
||||
// Memberships are where a person is.
|
||||
Memberships []store.Membership
|
||||
// Capacity is its capacity quota, nil when it has none.
|
||||
Capacity *capacityRow
|
||||
HasCapacity bool
|
||||
BudgetsBind bool
|
||||
|
||||
// Its use over the last week, and the totals beside it.
|
||||
HasUsage bool
|
||||
Requests, Tokens, Spend TimeSeries
|
||||
Totals usageRow
|
||||
Priced bool
|
||||
UsageWindow, QuotaWindow string
|
||||
|
||||
CSRF string
|
||||
}
|
||||
|
||||
// renderPeople draws the people page on the tab asked for.
|
||||
func (p *AccountPages) renderPeople(w http.ResponseWriter, r *http.Request, admin store.User, v peopleView) {
|
||||
d := peopleData{
|
||||
Nav: navFrom(r, "people"), Tab: v.Tab,
|
||||
HasPeople: p.peopleStore != nil, HasLimits: p.quotaStore != nil, HasUsage: p.usageStore != nil,
|
||||
Sel: r.FormValue("sel"),
|
||||
Version: p.version, Now: time.Now().UTC().Format(time.RFC3339),
|
||||
}
|
||||
switch {
|
||||
case d.Tab == tabPeople && d.HasPeople:
|
||||
u := p.buildUsers(r, p.peopleStore, admin, v.Msg, v.Err, v.Open)
|
||||
d.Users, d.Pending = &u, len(u.Pending)
|
||||
if d.Sel != "" {
|
||||
d.Detail = p.detail(r, d.Sel, &u)
|
||||
}
|
||||
case d.Tab == tabLimits && d.HasLimits:
|
||||
q := p.buildQuotas(r, p.quotaStore, v.Msg, v.Err)
|
||||
d.Quotas = &q
|
||||
case d.Tab == tabUsage && d.HasUsage:
|
||||
u := p.buildUsage(r, p.usageStore, p.usagePriced)
|
||||
d.Usage = &u
|
||||
default:
|
||||
// The tab asked for is not mounted here: the first one that is.
|
||||
switch {
|
||||
case d.HasPeople:
|
||||
v.Tab = tabPeople
|
||||
case d.HasLimits:
|
||||
v.Tab = tabLimits
|
||||
case d.HasUsage:
|
||||
v.Tab = tabUsage
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
p.renderPeople(w, r, admin, v)
|
||||
return
|
||||
}
|
||||
render(w, "admin_people", d)
|
||||
}
|
||||
|
||||
// usageGrouping is the grouping a branch's or a person's use is read under.
|
||||
func usageGrouping(kind store.Scope) (store.Grouping, bool) {
|
||||
switch kind {
|
||||
case store.ScopeOrg:
|
||||
return store.GroupByOrg, true
|
||||
case store.ScopeTeam:
|
||||
return store.GroupByTeam, true
|
||||
case store.ScopeProject:
|
||||
return store.GroupByProject, true
|
||||
case store.ScopeUser:
|
||||
return store.GroupByUser, true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// detail gathers what the page knows about one branch or person.
|
||||
func (p *AccountPages) detail(r *http.Request, sel string, u *adminUsersData) *peopleDetail {
|
||||
kind, id, ok := strings.Cut(sel, ":")
|
||||
if !ok || id == "" {
|
||||
return nil
|
||||
}
|
||||
ctx := r.Context()
|
||||
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfFor(r),
|
||||
UsageWindow: "last 7 days", QuotaWindow: "30 days"}
|
||||
if !p.nameDetail(d, u) {
|
||||
return nil
|
||||
}
|
||||
|
||||
if p.quotaStore != nil {
|
||||
d.BudgetsBind = p.priced != nil && p.priced.AnythingPriced()
|
||||
p.limitsDetail(ctx, d)
|
||||
}
|
||||
if p.capacity != nil && d.Kind != store.ScopeUser {
|
||||
d.HasCapacity = true
|
||||
if views, err := p.capacity.QuotaViews(ctx); err == nil {
|
||||
for _, v := range views {
|
||||
if v.Scope == sel {
|
||||
c := capacityRow{QuotaView: v,
|
||||
GPUs: capCell(v.Holds.GPUs, v.Guarantee.GPUs, v.Limit.GPUs),
|
||||
Cores: capCell(v.Holds.CPUCores, v.Guarantee.CPUCores, v.Limit.CPUCores),
|
||||
Memory: capCell(v.Holds.MemoryGB, v.Guarantee.MemoryGB, v.Limit.MemoryGB)}
|
||||
d.Capacity = &c
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if p.usageStore != nil {
|
||||
p.usageDetail(ctx, d)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// nameDetail finds what was chosen in the tree or among the accounts, and
|
||||
// says whether it is there at all.
|
||||
func (p *AccountPages) nameDetail(d *peopleDetail, u *adminUsersData) bool {
|
||||
if d.Kind == store.ScopeUser {
|
||||
for _, a := range u.Accounts {
|
||||
if a.ID == d.ID {
|
||||
d.Name, d.Path, d.Memberships = a.Email, a.Name, a.Memberships
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
for _, o := range u.Tree {
|
||||
if d.Kind == store.ScopeOrg && o.ID == d.ID {
|
||||
d.Name, d.Path = o.Name, o.Name
|
||||
return true
|
||||
}
|
||||
for _, t := range o.Teams {
|
||||
if d.Kind == store.ScopeTeam && t.ID == d.ID {
|
||||
d.Name, d.Path = t.Name, o.Name+" / "+t.Name
|
||||
return true
|
||||
}
|
||||
for _, pr := range t.Projects {
|
||||
if d.Kind == store.ScopeProject && pr.ID == d.ID {
|
||||
d.Name, d.Path = pr.Name, o.Name+" / "+t.Name+" / "+pr.Name
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// limitsDetail is the branch's own quota row, or the person's own limit.
|
||||
func (p *AccountPages) limitsDetail(ctx context.Context, d *peopleDetail) {
|
||||
s := p.quotaStore
|
||||
limits, err := s.AllLimits(ctx)
|
||||
if err != nil {
|
||||
p.log.Warn("could not read the limits for a detail panel", "err", err)
|
||||
return
|
||||
}
|
||||
since := time.Now().Add(-quotaWindow)
|
||||
if d.Kind == store.ScopeUser {
|
||||
for _, row := range p.personRows(ctx, s, limits, since, "") {
|
||||
if row.ID == d.ID {
|
||||
row := row
|
||||
d.Person = &row
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
tree, err := s.Tree(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
for _, row := range quotaRows(tree, limits, p.spendByScope(ctx, s, since)) {
|
||||
if row.Scope == d.Kind && row.ID == d.ID {
|
||||
ms := store.MemberScope(row.Scope)
|
||||
if l, ok := limits[store.LimitOwner{Scope: ms, ID: row.ID}.CounterKey()]; ok {
|
||||
row.MemberRaw, row.Member = fieldsOf(l), describeLimits(l)
|
||||
}
|
||||
d.Limits = &row
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// usageDetail is the branch's or person's use over the last week, drawn.
|
||||
func (p *AccountPages) usageDetail(ctx context.Context, d *peopleDetail) {
|
||||
g, ok := usageGrouping(d.Kind)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
win := windowFor("7d")
|
||||
rows, err := p.usageStore.UsageBucketsBy(ctx, g, time.Now().Add(-win.Span), win.Width)
|
||||
if err != nil {
|
||||
p.log.Warn("could not read usage for a detail panel", "sel", string(d.Kind)+":"+d.ID, "err", err)
|
||||
return
|
||||
}
|
||||
d.HasUsage = true
|
||||
d.Priced = p.usagePriced != nil && p.usagePriced.AnythingPriced()
|
||||
for _, s := range rows {
|
||||
if s.ID == d.ID {
|
||||
var table []usageRow
|
||||
d.Requests, d.Tokens, d.Spend, table = usageCharts([]store.UsageSeries{s})
|
||||
d.Totals = table[0]
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// peoplePage mounts all three tabs, as the server does.
|
||||
func peoplePage(t *testing.T, q *fakeQuotas, u *fakeScopeSeries) *http.ServeMux {
|
||||
t.Helper()
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsers(mux, &fakeAdmin{orgs: q.orgs})
|
||||
p.RegisterAdminQuotas(mux, q, pricedYes{})
|
||||
p.RegisterAdminUsage(mux, u, pricedYes{})
|
||||
return mux
|
||||
}
|
||||
|
||||
func TestABranchChosenInTheTreeShowsItsLimitsAndUseTogether(t *testing.T) {
|
||||
// People, their limits and their use were three pages: a team's members
|
||||
// on one, its quota on another, its spend on a third.
|
||||
q := oneTeam()
|
||||
rpm := 60
|
||||
q.limits["org:org-1"] = store.Limits{Scope: store.ScopeOrg, ScopeID: "org-1", RPM: &rpm}
|
||||
u := &fakeScopeSeries{byGroup: map[store.Grouping][]store.UsageSeries{
|
||||
store.GroupByOrg: {{Grouping: store.GroupByOrg, ID: "org-1", Name: "Scania", Requests: 1234,
|
||||
Buckets: []store.Bucket{{Start: time.Now().Add(-time.Hour), Requests: 1234}}}},
|
||||
}}
|
||||
body := strings.Join(strings.Fields(
|
||||
mirrorGet(t, peoplePage(t, q, u), "/admin/users?sel=org:org-1").Body.String()), " ")
|
||||
|
||||
for _, want := range []string{
|
||||
`class="people-detail"`, "<h2>Scania</h2>",
|
||||
"<dt>Requests/min</dt><dd>60</dd>", // its own limit
|
||||
"1.2k requests", // its use over the week
|
||||
`name="sel" value="org:org-1"`, // changing it comes back here
|
||||
`aria-current="page">Organizations & people`,
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the panel lacks %q", want)
|
||||
}
|
||||
}
|
||||
if u.asked[len(u.asked)-1] != store.GroupByOrg {
|
||||
t.Errorf("use was read by %v, not by organization", u.asked)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAChangeMadeInAPanelComesBackToThePanel(t *testing.T) {
|
||||
q := oneTeam()
|
||||
body := mirrorPost(t, peoplePage(t, q, &fakeScopeSeries{}), "/admin/quotas", url.Values{
|
||||
"csrf": {"tok"}, "scope": {"org"}, "id": {"org-1"}, "rpm": {"30"},
|
||||
"tab": {"people"}, "sel": {"org:org-1"},
|
||||
}).Body.String()
|
||||
if len(q.put) != 1 || *q.put[0].RPM != 30 {
|
||||
t.Fatalf("the limit was not saved: %+v", q.put)
|
||||
}
|
||||
if !strings.Contains(body, `class="people-detail"`) || !strings.Contains(body, "rpm 30") {
|
||||
t.Error("saving from the panel did not come back to it with the result")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachOldAddressOpensItsTab(t *testing.T) {
|
||||
mux := peoplePage(t, oneTeam(), &fakeScopeSeries{})
|
||||
for path, tab := range map[string]string{
|
||||
"/admin/users": "Organizations & people", "/admin/quotas": "Limits", "/admin/usage": "Usage",
|
||||
} {
|
||||
body := mirrorGet(t, mux, path).Body.String()
|
||||
if !strings.Contains(body, `aria-current="page">`+tab) {
|
||||
t.Errorf("%s does not open the %s tab", path, tab)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSomethingNotInTheTreeOpensNoPanel(t *testing.T) {
|
||||
body := mirrorGet(t, peoplePage(t, oneTeam(), &fakeScopeSeries{}), "/admin/users?sel=org:gone").Body.String()
|
||||
if strings.Contains(body, `class="people-detail"`) {
|
||||
t.Error("a panel opened for an organization that does not exist")
|
||||
}
|
||||
}
|
||||
@@ -159,10 +159,11 @@ type adminQuotasData struct {
|
||||
func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, priced Priced) {
|
||||
p.hasQuotas = true
|
||||
p.priced = priced
|
||||
p.quotaStore = s
|
||||
|
||||
mux.HandleFunc("GET /admin/quotas", p.adminOnly(
|
||||
func(w http.ResponseWriter, r *http.Request, _ store.User) {
|
||||
p.renderQuotas(w, r, s, "", "")
|
||||
func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
p.renderPeople(w, r, admin, peopleView{Tab: tabLimits})
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/quotas", p.adminOnly(
|
||||
@@ -177,10 +178,10 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
|
||||
p.log.Warn("a quota change was refused",
|
||||
"admin", admin.Email, "scope", r.FormValue("scope"),
|
||||
"id", r.FormValue("id"), "err", err)
|
||||
p.renderQuotas(w, r, s, "", err.Error())
|
||||
p.renderQuotas(w, r, admin, "", err.Error())
|
||||
return
|
||||
}
|
||||
p.renderQuotas(w, r, s, msg, "")
|
||||
p.renderQuotas(w, r, admin, msg, "")
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/quotas/capacity", p.adminOnly(
|
||||
@@ -194,11 +195,11 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
|
||||
if err != nil {
|
||||
p.log.Warn("a capacity quota change was refused",
|
||||
"admin", admin.Email, "scope", r.FormValue("scope"), "err", err)
|
||||
p.renderQuotas(w, r, s, "", err.Error())
|
||||
p.renderQuotas(w, r, admin, "", err.Error())
|
||||
return
|
||||
}
|
||||
p.log.Info("capacity quota changed", "admin", admin.Email, "what", msg)
|
||||
p.renderQuotas(w, r, s, msg, "")
|
||||
p.renderQuotas(w, r, admin, msg, "")
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -270,10 +271,24 @@ func (p *AccountPages) applyQuota(r *http.Request, s QuotaAdmin, admin store.Use
|
||||
return string(scope) + " " + id + ": " + now, nil
|
||||
}
|
||||
|
||||
// renderQuotas answers a change to a limit: on the limits tab, or on the
|
||||
// people tab when it was made from a branch's own panel there.
|
||||
func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
|
||||
s QuotaAdmin, msg, errMsg string,
|
||||
admin store.User, msg, errMsg string,
|
||||
) {
|
||||
const window = 30 * 24 * time.Hour
|
||||
tab := tabLimits
|
||||
if r.FormValue("tab") == tabPeople {
|
||||
tab = tabPeople
|
||||
}
|
||||
p.renderPeople(w, r, admin, peopleView{Tab: tab, Msg: msg, Err: errMsg})
|
||||
}
|
||||
|
||||
// quotaWindow is how far back spend against a budget is counted.
|
||||
const quotaWindow = 30 * 24 * time.Hour
|
||||
|
||||
// buildQuotas is every limit, as the limits tab shows them.
|
||||
func (p *AccountPages) buildQuotas(r *http.Request, s QuotaAdmin, msg, errMsg string) adminQuotasData {
|
||||
const window = quotaWindow
|
||||
d := adminQuotasData{
|
||||
Nav: navFrom(r, "quotas"),
|
||||
Since: "30 days", Message: msg, Error: errMsg,
|
||||
@@ -289,8 +304,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
|
||||
if err != nil {
|
||||
p.log.Error("could not read the limits", "err", err)
|
||||
d.Error = "could not read the limits"
|
||||
render(w, "admin_quotas", d)
|
||||
return
|
||||
return d
|
||||
}
|
||||
spend := p.spendByScope(r.Context(), s, time.Now().Add(-window))
|
||||
|
||||
@@ -314,7 +328,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
|
||||
d.Rows[i].Editing = d.Editing ==
|
||||
string(d.Rows[i].Scope)+":"+d.Rows[i].ID
|
||||
}
|
||||
render(w, "admin_quotas", d)
|
||||
return d
|
||||
}
|
||||
|
||||
// spendByScope is what each group has used in the window, keyed like a limit.
|
||||
|
||||
+182
-6
@@ -43,8 +43,20 @@ type serveForm struct {
|
||||
Engine string
|
||||
// Profile is the profile name, which doubles as how a node_selector would
|
||||
// be written later. Named for the hardware rather than the model, because
|
||||
// a profile is about where a model runs.
|
||||
// a profile is about where a model runs. Derived from the placement, not
|
||||
// typed: a name is only a name, and a field for it read as though calling
|
||||
// a profile "gpu" would put it on one.
|
||||
Profile string
|
||||
// Placements are where it can run: each GPU class the fleet has, and for
|
||||
// llama.cpp the CPU. One is selected.
|
||||
Placements []placeChoice
|
||||
// GPUs is how many cards a replica takes, when placed on a GPU class.
|
||||
GPUs int
|
||||
// Parallel is how many requests one replica serves at once.
|
||||
Parallel int
|
||||
// FreeGPUs is how many cards no replica holds, across the fleet: what
|
||||
// choosing the CPU leaves unused.
|
||||
FreeGPUs int
|
||||
// Files is how many files the version holds, above one for a set.
|
||||
Files int
|
||||
// MinMemoryGB is the host RAM to ask for, estimated from the weights.
|
||||
@@ -126,6 +138,13 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
|
||||
default:
|
||||
f.Profile = "gpu"
|
||||
}
|
||||
f.GPUs, f.Parallel = 1, defaultParallel
|
||||
f.Placements, f.FreeGPUs = placementsFor(fleet, f.Engine)
|
||||
for _, pc := range f.Placements {
|
||||
if pc.Selected {
|
||||
f.Profile = pc.profile()
|
||||
}
|
||||
}
|
||||
// Whether any node can actually run it. Agents report their installed
|
||||
// engines now, so this is a fact rather than the guess it started as —
|
||||
// which was cards, and would have passed a GPU node with no vLLM on it.
|
||||
@@ -274,12 +293,38 @@ func buildModel(get func(string) string) (catalog.Model, error) {
|
||||
if engine == "" {
|
||||
return m, errors.New("which engine should run it?")
|
||||
}
|
||||
p := catalog.Profile{
|
||||
Name: strings.TrimSpace(get("profile")), Engine: engine,
|
||||
Weight: 1, Priority: 100,
|
||||
p := catalog.Profile{Engine: engine, Weight: 1, Priority: 100}
|
||||
gpus := 1
|
||||
if v := strings.TrimSpace(get("gpus")); v != "" {
|
||||
if gpus, err = positive(v, "the GPUs per replica"); err != nil {
|
||||
return m, err
|
||||
}
|
||||
gpus = max(gpus, 1)
|
||||
}
|
||||
if p.Name == "" {
|
||||
p.Name = engine
|
||||
place := placeChoice{Value: strings.TrimSpace(get("placement"))}
|
||||
switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
|
||||
case onGPU && class != "":
|
||||
// Placement is decided by gpu_class, which is what the selector
|
||||
// sets; the count is gpus_per_node for llama.cpp, which spreads
|
||||
// layers over a node's cards, and tp for the others.
|
||||
p.GPUClass = class
|
||||
if engine == "llamacpp" {
|
||||
p.GPUsPerNode = gpus
|
||||
} else {
|
||||
p.TP = gpus
|
||||
}
|
||||
place.GPUs = gpus
|
||||
case place.Value == "" || place.Value == "cpu":
|
||||
if engine != "llamacpp" {
|
||||
return m, fmt.Errorf("%s runs on GPUs; choose a GPU class to place it on", engine)
|
||||
}
|
||||
place.Value = "cpu"
|
||||
default:
|
||||
return m, fmt.Errorf("%q is not a placement", place.Value)
|
||||
}
|
||||
p.Name = place.profile()
|
||||
if v := strings.TrimSpace(get("profile")); v != "" {
|
||||
p.Name = v
|
||||
}
|
||||
// The pool, not the per-request limit: llama.cpp divides it across its
|
||||
// parallel slots, so the number a client may use is smaller and is what
|
||||
@@ -290,6 +335,15 @@ func buildModel(get func(string) string) (catalog.Model, error) {
|
||||
}
|
||||
p.Args = []string{"--ctx-size", v}
|
||||
}
|
||||
if v := strings.TrimSpace(get("parallel")); v != "" && engine == "llamacpp" {
|
||||
n, err := positive(v, "the parallel requests")
|
||||
if err != nil {
|
||||
return m, err
|
||||
}
|
||||
if n > 0 {
|
||||
p.Args = append(p.Args, "--parallel", strconv.Itoa(n))
|
||||
}
|
||||
}
|
||||
if v := strings.TrimSpace(get("min_memory_gb")); v != "" {
|
||||
if p.MinMemoryGB, err = positive(v, "the memory requirement"); err != nil {
|
||||
return m, err
|
||||
@@ -408,3 +462,125 @@ func modeOf(sc catalog.Scaling) string {
|
||||
// defaultCtxPool is the llama.cpp context pool the serve form starts with.
|
||||
// --parallel divides it among slots; it does not multiply it.
|
||||
const defaultCtxPool = 16384
|
||||
|
||||
// defaultParallel is the requests a llama.cpp replica serves at once when
|
||||
// the form is not changed: the agent's own default.
|
||||
const defaultParallel = 4
|
||||
|
||||
// placeChoice is one place a model can run: a GPU class, or the CPU.
|
||||
type placeChoice struct {
|
||||
// Value is "gpu:<class>" or "cpu", as the form submits it.
|
||||
Value string
|
||||
Label string
|
||||
// GPUs is how many cards a replica takes, for naming the profile.
|
||||
GPUs int
|
||||
Selected bool
|
||||
}
|
||||
|
||||
// profile names the profile for where it runs: the class, with the count when
|
||||
// it is more than one card ("t4", "t4-x4"), or "cpu".
|
||||
func (c placeChoice) profile() string {
|
||||
class, ok := strings.CutPrefix(c.Value, "gpu:")
|
||||
if !ok || class == "" {
|
||||
return "cpu"
|
||||
}
|
||||
if c.GPUs > 1 {
|
||||
return fmt.Sprintf("%s-x%d", class, c.GPUs)
|
||||
}
|
||||
return class
|
||||
}
|
||||
|
||||
// placementsFor lists where engine can run on this fleet, with one selected,
|
||||
// and how many GPUs are free.
|
||||
//
|
||||
// A GPU when the fleet has one: a GGUF model was put on the CPU by this form
|
||||
// on a cluster of GPUs, ran at a tenth of their speed, and nothing on the way
|
||||
// said why — the form had no choice to offer and named the profile "cpu". The
|
||||
// class selected is the one with the most cards free.
|
||||
func placementsFor(fleet Source, engine string) ([]placeChoice, int) {
|
||||
var out []placeChoice
|
||||
free, best := 0, -1
|
||||
if fleet != nil {
|
||||
for _, c := range fleet.Summary().GPUs {
|
||||
n := c.Free()
|
||||
free += n
|
||||
out = append(out, placeChoice{
|
||||
Value: "gpu:" + c.Class,
|
||||
Label: fmt.Sprintf("GPU: %s, %d of %d free", c.Class, n, c.Total),
|
||||
})
|
||||
if best < 0 || n > fleet.Summary().GPUs[best].Free() {
|
||||
best = len(out) - 1
|
||||
}
|
||||
}
|
||||
}
|
||||
if best >= 0 {
|
||||
out[best].Selected = true
|
||||
}
|
||||
if engine == "llamacpp" {
|
||||
label := "CPU only: no GPU, many times slower"
|
||||
if len(out) == 0 {
|
||||
label = "CPU: this fleet has no GPU"
|
||||
}
|
||||
out = append(out, placeChoice{Value: "cpu", Label: label, Selected: best < 0})
|
||||
}
|
||||
return out, free
|
||||
}
|
||||
|
||||
// freeGPUs is how many cards no replica holds across the fleet.
|
||||
func freeGPUs(fleet Source) int {
|
||||
if fleet == nil {
|
||||
return 0
|
||||
}
|
||||
n := 0
|
||||
for _, c := range fleet.Summary().GPUs {
|
||||
n += c.Free()
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// describeServe says, in one line, what a model will run as: the summary
|
||||
// shown before it is added and in the message after.
|
||||
func describeServe(m catalog.Model) string {
|
||||
if len(m.Profiles) == 0 {
|
||||
return ""
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
engine := map[string]string{"llamacpp": "llama.cpp", "vllm": "vLLM", "sglang": "SGLang"}[p.Engine]
|
||||
if engine == "" {
|
||||
engine = p.Engine
|
||||
}
|
||||
where := "on the CPU"
|
||||
if p.GPUClass != "" {
|
||||
n := max(p.GPUsPerNode, p.TP, 1)
|
||||
where = fmt.Sprintf("on %s %s", plural(n, "GPU", "GPUs"), p.GPUClass)
|
||||
if n > 1 {
|
||||
where = fmt.Sprintf("on %d %s GPUs of one node", n, p.GPUClass)
|
||||
}
|
||||
}
|
||||
parts := []string{engine + " " + where}
|
||||
if p.MinMemoryGB > 0 {
|
||||
parts = append(parts, fmt.Sprintf("%d GB of host memory", p.MinMemoryGB))
|
||||
}
|
||||
if p.GPUClass == "" && p.CPUCores > 0 {
|
||||
parts = append(parts, plural(p.CPUCores, "core", "cores"))
|
||||
}
|
||||
pool, parallel := argInt(p.Args, "--ctx-size"), argInt(p.Args, "--parallel")
|
||||
switch {
|
||||
case pool > 0 && parallel > 0:
|
||||
parts = append(parts, fmt.Sprintf("context pool %d over %d parallel requests (%d each)", pool, parallel, pool/parallel))
|
||||
case pool > 0:
|
||||
parts = append(parts, fmt.Sprintf("context pool %d", pool))
|
||||
}
|
||||
return strings.Join(parts, ", ")
|
||||
}
|
||||
|
||||
// argInt is the number after flag in args, or 0.
|
||||
func argInt(args []string, flag string) int {
|
||||
for i := 0; i+1 < len(args); i++ {
|
||||
if args[i] == flag {
|
||||
n, _ := strconv.Atoi(args[i+1])
|
||||
return n
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -71,6 +71,8 @@ type fakeFleet struct {
|
||||
gpus int
|
||||
engines []string
|
||||
memoryGB int
|
||||
// used is how many of the cards replicas hold.
|
||||
used int
|
||||
}
|
||||
|
||||
func (f fakeFleet) Nodes() []adminapi.NodeView {
|
||||
@@ -81,7 +83,12 @@ func (f fakeFleet) Nodes() []adminapi.NodeView {
|
||||
func (f fakeFleet) Models() []adminapi.ModelView { return nil }
|
||||
func (f fakeFleet) Replicas(string) []adminapi.ReplicaView { return nil }
|
||||
func (f fakeFleet) Explain(string) adminapi.ExplainView { return adminapi.ExplainView{} }
|
||||
func (f fakeFleet) Summary() adminapi.Summary { return adminapi.Summary{} }
|
||||
func (f fakeFleet) Summary() adminapi.Summary {
|
||||
if f.gpus == 0 {
|
||||
return adminapi.Summary{}
|
||||
}
|
||||
return adminapi.Summary{GPUs: []adminapi.ClassUse{{Class: "t4", Total: f.gpus, Used: f.used}}}
|
||||
}
|
||||
|
||||
func held(model, version, file string) mirror.Status {
|
||||
return mirror.Status{Model: model, Version: version, File: file,
|
||||
@@ -452,3 +459,90 @@ func TestServingOnDemandLoadsWhenAskedFor(t *testing.T) {
|
||||
t.Errorf("on demand was added as %+v", sc)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGGUFOnAFleetWithGPUsIsOfferedTheGPUFirst(t *testing.T) {
|
||||
gguf := []mirror.Status{held("coder", "v1", "coder-q4_k_m.gguf")}
|
||||
body := mirrorGet(t, servePage(t, gguf, &fakeCatalog{}, 4, "llamacpp"),
|
||||
"/admin/mirror?serve=coder:v1").Body.String()
|
||||
// Found in the field: no choice offered, a profile called "cpu", and a
|
||||
// model on the CPU of a GPU cluster at a tenth of the speed.
|
||||
if !strings.Contains(body, `<option value="gpu:t4" selected>GPU: t4, 4 of 4 free</option>`) {
|
||||
t.Errorf("the free GPU class should be offered and chosen; got %s", body)
|
||||
}
|
||||
if !strings.Contains(body, `<option value="cpu">CPU only`) {
|
||||
t.Error("the CPU should still be on offer, unselected")
|
||||
}
|
||||
if !strings.Contains(body, `name="cpu_ok"`) {
|
||||
t.Error("choosing the CPU with GPUs free needs saying so")
|
||||
}
|
||||
|
||||
cpuOnly := mirrorGet(t, servePage(t, gguf, &fakeCatalog{}, 0, "llamacpp"),
|
||||
"/admin/mirror?serve=coder:v1").Body.String()
|
||||
if !strings.Contains(cpuOnly, `<option value="cpu" selected>CPU: this fleet has no GPU</option>`) ||
|
||||
strings.Contains(cpuOnly, `name="cpu_ok"`) {
|
||||
t.Errorf("a fleet without GPUs has only the CPU, and nothing to confirm; got %s", cpuOnly)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheChosenPlacementDecidesWhereItRuns(t *testing.T) {
|
||||
form := func(engine, place, gpus string) url.Values {
|
||||
return url.Values{
|
||||
"name": {"m"}, "source": {"http://x/m"}, "engine": {engine}, "replicas": {"1"},
|
||||
"context_length": {"4096"}, "ctx_size": {"16384"}, "parallel": {"4"},
|
||||
"placement": {place}, "gpus": {gpus}, "min_memory_gb": {"8"},
|
||||
}
|
||||
}
|
||||
m, err := buildModel(form("llamacpp", "gpu:t4", "4").Get)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
if p.GPUClass != "t4" || p.GPUsPerNode != 4 || p.TP > 1 || p.Name != "t4-x4" {
|
||||
t.Errorf("llama.cpp on four t4s: %+v", p)
|
||||
}
|
||||
if got := describeServe(m); got != "llama.cpp on 4 t4 GPUs of one node, 8 GB of host memory, "+
|
||||
"context pool 16384 over 4 parallel requests (4096 each)" {
|
||||
t.Errorf("summary %q", got)
|
||||
}
|
||||
if m, _ := buildModel(form("vllm", "gpu:t4", "2").Get); m.Profiles[0].TP != 2 || m.Profiles[0].GPUsPerNode != 0 {
|
||||
t.Errorf("vLLM's GPU count is tp: %+v", m.Profiles[0])
|
||||
}
|
||||
if m, _ := buildModel(form("llamacpp", "cpu", "1").Get); m.Profiles[0].GPUClass != "" || m.Profiles[0].Name != "cpu" {
|
||||
t.Errorf("the CPU: %+v", m.Profiles[0])
|
||||
}
|
||||
if _, err := buildModel(form("vllm", "cpu", "1").Get); err == nil {
|
||||
t.Error("vLLM on the CPU was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCPUWithGPUsFreeIsAChoiceMadeOnPurpose(t *testing.T) {
|
||||
values := func(extra ...string) url.Values {
|
||||
v := url.Values{
|
||||
"csrf": {"tok"}, "name": {"coder"}, "source": {reachableSource(t)}, "engine": {"llamacpp"},
|
||||
"replicas": {"1"}, "context_length": {"4096"}, "placement": {"cpu"}, "min_memory_gb": {"8"},
|
||||
}
|
||||
for i := 0; i+1 < len(extra); i += 2 {
|
||||
v.Set(extra[i], extra[i+1])
|
||||
}
|
||||
return v
|
||||
}
|
||||
cat := &fakeCatalog{}
|
||||
page := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat,
|
||||
Fleet: fakeFleet{gpus: 4, used: 1, engines: []string{"llamacpp"}}}, true)
|
||||
body := mirrorPost(t, page, "/admin/mirror/serve", values()).Body.String()
|
||||
if len(cat.added) != 0 || !strings.Contains(body, "3 GPUs free") {
|
||||
t.Fatalf("the CPU with three GPUs free went through unasked: %s", body)
|
||||
}
|
||||
mirrorPost(t, page, "/admin/mirror/serve", values("cpu_ok", "yes"))
|
||||
if len(cat.added) != 1 {
|
||||
t.Fatal("ticking the box should add it")
|
||||
}
|
||||
// Every card held: nothing to ask.
|
||||
cat = &fakeCatalog{}
|
||||
full := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat,
|
||||
Fleet: fakeFleet{gpus: 4, used: 4, engines: []string{"llamacpp"}}}, true)
|
||||
mirrorPost(t, full, "/admin/mirror/serve", values())
|
||||
if len(cat.added) != 1 {
|
||||
t.Error("with no GPU free the CPU is the only choice and should not be questioned")
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
@@ -80,6 +81,12 @@ type adminUsageData struct {
|
||||
// chart along the bottom reads as "nobody spent anything".
|
||||
Priced bool
|
||||
|
||||
// GPUHeld and GPUHours are the cards each owner's models held over the
|
||||
// window, from the metrics store; empty without one, or with no owned
|
||||
// model on a GPU.
|
||||
GPUHeld TimeSeries
|
||||
GPUHours []gpuHoursRow
|
||||
|
||||
Error string
|
||||
Version string
|
||||
Now string
|
||||
@@ -92,15 +99,15 @@ type adminUsageData struct {
|
||||
// answers "who is using the cluster, and when" — including per person, which is
|
||||
// not a level quotas can be written on at all.
|
||||
func (p *AccountPages) RegisterAdminUsage(mux *http.ServeMux, s UsageAdmin, priced Priced) {
|
||||
mux.HandleFunc("GET /admin/usage", p.AdminPage(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
p.renderUsage(w, r, s, priced)
|
||||
p.usageStore, p.usagePriced = s, priced
|
||||
mux.HandleFunc("GET /admin/usage", p.adminOnly(
|
||||
func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
p.renderPeople(w, r, admin, peopleView{Tab: tabUsage})
|
||||
}))
|
||||
}
|
||||
|
||||
func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
|
||||
s UsageAdmin, priced Priced,
|
||||
) {
|
||||
// buildUsage is the cluster's use over time, as the usage tab shows it.
|
||||
func (p *AccountPages) buildUsage(r *http.Request, s UsageAdmin, priced Priced) adminUsageData {
|
||||
by := store.Grouping(r.URL.Query().Get("by"))
|
||||
if !by.Valid() {
|
||||
by = store.GroupByTeam
|
||||
@@ -130,12 +137,48 @@ func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
|
||||
if err != nil {
|
||||
p.log.Error("could not read usage over time", "by", by, "err", err)
|
||||
d.Error = "could not read usage"
|
||||
render(w, "admin_usage", d)
|
||||
return
|
||||
return d
|
||||
}
|
||||
|
||||
d.Requests, d.Tokens, d.Spend, d.Rows = usageCharts(rows)
|
||||
render(w, "admin_usage", d)
|
||||
d.GPUHeld, d.GPUHours = p.gpuHours(r.Context(), win)
|
||||
return d
|
||||
}
|
||||
|
||||
// gpuHoursRow is the GPU time one owner's models held over the window.
|
||||
type gpuHoursRow struct {
|
||||
Owner string
|
||||
Hours float64
|
||||
// Now is how many cards they hold at the end of the window.
|
||||
Now float64
|
||||
}
|
||||
|
||||
// gpuHours is the cards held by each owner's models over the window, drawn
|
||||
// and summed into GPU-hours. From the metrics store, because what a model
|
||||
// holds is placement's, not the request log's: a model nobody asks still
|
||||
// holds its cards. As far back as the store keeps history, which for a
|
||||
// default Prometheus is fifteen days.
|
||||
func (p *AccountPages) gpuHours(ctx context.Context, win usageWindow) (TimeSeries, []gpuHoursRow) {
|
||||
if p.history == nil {
|
||||
return TimeSeries{}, nil
|
||||
}
|
||||
series, err := p.history.RangeQuery(ctx, `sum by (owner) (llmgw_gpu_assigned)`, "owner", win.Span, win.Width)
|
||||
if err != nil || len(series) == 0 {
|
||||
return TimeSeries{}, nil
|
||||
}
|
||||
var rows []gpuHoursRow
|
||||
for i := range series {
|
||||
if series[i].Name == "" {
|
||||
series[i].Name = "no owner"
|
||||
}
|
||||
row := gpuHoursRow{Owner: series[i].Name, Now: series[i].Last()}
|
||||
// Each point is the cards held at that instant, standing for the
|
||||
// step that follows it.
|
||||
row.Hours = series[i].Sum() * win.Width.Hours()
|
||||
rows = append(rows, row)
|
||||
}
|
||||
sort.Slice(rows, func(i, j int) bool { return rows[i].Hours > rows[j].Hours })
|
||||
return LineChart(series, "", 6), rows
|
||||
}
|
||||
|
||||
// seriesOf turns each group's buckets into a line of one measure.
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -162,3 +163,32 @@ func TestAPersonWithNoAccountIsNamedRatherThanBlank(t *testing.T) {
|
||||
t.Errorf("unowned traffic was left unlabelled; got %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheUsagePageCountsTheGPUTimeEachOwnerHeld(t *testing.T) {
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
// Over a day in 20-minute steps: team:research held 8 cards throughout,
|
||||
// nobody's model 2 for the last three steps.
|
||||
res, none := promq.Series{Name: "team:research"}, promq.Series{Name: ""}
|
||||
for i := range 72 {
|
||||
res.Points = append(res.Points, promq.Point{Value: 8})
|
||||
if i >= 69 {
|
||||
none.Points = append(none.Points, promq.Point{Value: 2})
|
||||
}
|
||||
}
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{`sum by (owner) (llmgw_gpu_assigned)`: {res, none}}}
|
||||
p.SetHistory(h)
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsage(mux, &fakeScopeSeries{}, nil)
|
||||
body := mirrorGet(t, mux, "/admin/usage?window=24h").Body.String()
|
||||
for _, want := range []string{"GPU time, by owner", "<td>team:research</td><td class=\"num\">192.0</td>",
|
||||
"<td>no owner</td><td class=\"num\">2.0</td>"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// Without a metrics store, nothing is said about GPU time.
|
||||
if body := mirrorGet(t, usagePage(t, &fakeScopeSeries{}, nil), "/admin/usage").Body.String(); strings.Contains(body, "GPU time") {
|
||||
t.Error("GPU time shown with no metrics store to read it from")
|
||||
}
|
||||
}
|
||||
@@ -87,6 +87,9 @@ type adminUsersData struct {
|
||||
Projects []store.ProjectPath
|
||||
Orgs []store.Org
|
||||
Teams []store.TeamPath
|
||||
// Sel is the branch or person whose panel is open, "team:<id>", so the
|
||||
// tree can mark it.
|
||||
Sel string
|
||||
// Open is the project whose section is shown expanded: the one just acted
|
||||
// on, so the result of a change is in view without the reader hunting for
|
||||
// it. Scripted pages keep every open section open; this is for the rest.
|
||||
@@ -115,6 +118,7 @@ type adminUsersData struct {
|
||||
// shell is no hardship. Revoking a key is here, because it only ever takes
|
||||
// access away.
|
||||
func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
|
||||
p.peopleStore = s
|
||||
mux.HandleFunc("GET /admin/users", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
p.renderUsers(w, r, s, admin, "", "", r.URL.Query().Get("open"))
|
||||
}))
|
||||
@@ -480,12 +484,17 @@ func (p *AccountPages) adminAction(s AccountAdmin,
|
||||
}
|
||||
|
||||
func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
|
||||
s AccountAdmin, admin store.User, msg, errMsg, open string,
|
||||
_ AccountAdmin, admin store.User, msg, errMsg, open string,
|
||||
) {
|
||||
p.renderPeople(w, r, admin, peopleView{Tab: tabPeople, Msg: msg, Err: errMsg, Open: open})
|
||||
}
|
||||
|
||||
// buildUsers is the tree and its people, as the first tab shows them.
|
||||
func (p *AccountPages) buildUsers(r *http.Request, s AccountAdmin, admin store.User, msg, errMsg, open string) adminUsersData {
|
||||
ctx := r.Context()
|
||||
d := adminUsersData{
|
||||
Nav: navFrom(r, "people"),
|
||||
Message: msg, Error: errMsg, Open: open, AdminID: admin.ID,
|
||||
Message: msg, Error: errMsg, Open: open, AdminID: admin.ID, Sel: r.FormValue("sel"),
|
||||
CSRF: csrfFor(r),
|
||||
Version: p.version,
|
||||
HasMirror: p.hasMirror,
|
||||
@@ -558,7 +567,7 @@ func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
|
||||
d.Scopes = append(d.Scopes, scopeOption{"project:" + pp.ID, pp.Path})
|
||||
}
|
||||
sort.Slice(d.Scopes, func(i, j int) bool { return d.Scopes[i].Label < d.Scopes[j].Label })
|
||||
render(w, "admin_users", d)
|
||||
return d
|
||||
}
|
||||
|
||||
func keyState(k store.Key, now time.Time) string {
|
||||
|
||||
@@ -18,6 +18,33 @@ type ChatModel struct {
|
||||
// offered anyway: the first message is what asks, so hiding it until it
|
||||
// runs meant it could never be started from here.
|
||||
OnDemand bool
|
||||
// Hardware is what the model's running replicas compute on: "cpu", or a
|
||||
// GPU class. Several when it runs on both, and empty when nothing runs
|
||||
// or nothing said.
|
||||
Hardware []string
|
||||
}
|
||||
|
||||
// Runs names the hardware for the picker: "GPU", "CPU", or "GPU + CPU" for a
|
||||
// model with replicas on both, where which one answers is the router's choice
|
||||
// and each reply says which it was.
|
||||
func (m ChatModel) Runs() string {
|
||||
var gpu, cpu bool
|
||||
for _, h := range m.Hardware {
|
||||
if h == "cpu" {
|
||||
cpu = true
|
||||
} else {
|
||||
gpu = true
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case gpu && cpu:
|
||||
return "GPU + CPU"
|
||||
case gpu:
|
||||
return "GPU"
|
||||
case cpu:
|
||||
return "CPU"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ChatContext is one thing the chat page can spend against: a project, or the
|
||||
|
||||
@@ -185,3 +185,19 @@ func TestTheTruncationNoticeIsNotPutInsideTheReply(t *testing.T) {
|
||||
t.Errorf("the reply is written as HTML other than from /chat/render (%d assignments)", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChatModelRunsNamesTheHardware(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
hw []string
|
||||
want string
|
||||
}{
|
||||
{nil, ""},
|
||||
{[]string{"cpu"}, "CPU"},
|
||||
{[]string{"6000-24g"}, "GPU"},
|
||||
{[]string{"cpu", "6000-24g"}, "GPU + CPU"},
|
||||
} {
|
||||
if got := (ChatModel{Hardware: c.hw}).Runs(); got != c.want {
|
||||
t.Errorf("%v: %q, want %q", c.hw, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -19,13 +19,23 @@ type DashboardData struct {
|
||||
// of a GPU table that would only say there are none.
|
||||
CPU cpuCapacity
|
||||
// Menu is the shared top menu; Nav below is what the cluster page adds.
|
||||
Menu Nav
|
||||
Nodes []adminapi.NodeView
|
||||
Models []adminapi.ModelView
|
||||
Replicas []adminapi.ReplicaView
|
||||
Unplaced []adminapi.ExplainView
|
||||
Usage []store.KeyUsage
|
||||
Summary adminapi.Summary
|
||||
Menu Nav
|
||||
// GPUMap is every card coloured by one figure, and IdleGPUs the replicas
|
||||
// holding cards that are doing nothing; both from the last heartbeats.
|
||||
GPUMap *gpuMap
|
||||
IdleGPUs []idleHolding
|
||||
// Efficiency is what each model gets from its cards, from the metrics
|
||||
// store; nil without one.
|
||||
Efficiency []modelEfficiency
|
||||
// Notice and Problem are what the last change made from this page did,
|
||||
// carried over the redirect back to it.
|
||||
Notice, Problem string
|
||||
Nodes []adminapi.NodeView
|
||||
Models []adminapi.ModelView
|
||||
Replicas []adminapi.ReplicaView
|
||||
Unplaced []adminapi.ExplainView
|
||||
Usage []store.KeyUsage
|
||||
Summary adminapi.Summary
|
||||
// Fabrics is the physical fabrics and what disagrees about them, from a
|
||||
// source that can say.
|
||||
Fabrics *fabrics.Report
|
||||
@@ -170,6 +180,9 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
Now: time.Now().UTC().Format(time.RFC3339),
|
||||
AuthEnabled: usage != nil,
|
||||
}
|
||||
if d.Menu.Admin {
|
||||
d.Notice, d.Problem = r.URL.Query().Get("notice"), r.URL.Query().Get("problem")
|
||||
}
|
||||
if src != nil {
|
||||
d.Nodes = src.Nodes()
|
||||
d.CPU = cpuCapacityOf(d.Nodes)
|
||||
@@ -192,9 +205,17 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
d.AnyFabrics = d.AnyFabrics || len(n.Fabrics) > 0
|
||||
}
|
||||
d.NodePrefix = sharedPrefix(ids)
|
||||
if d.AnyGPUs {
|
||||
d.GPUMap = buildGPUMap(d.Nodes, d.Replicas, r.URL.Query().Get("gpumap"), d.NodePrefix)
|
||||
d.IdleGPUs = idleHoldings(d.Nodes, d.Replicas, d.NodePrefix)
|
||||
}
|
||||
for _, m := range d.Models {
|
||||
d.AnyFallbacks = d.AnyFallbacks || len(m.Fallbacks) > 0
|
||||
}
|
||||
if d.AnyGPUs {
|
||||
d.Fleet.LoadGPU(r.Context(), nav.History, nav.Window)
|
||||
d.Efficiency = loadEfficiency(r.Context(), nav.History)
|
||||
}
|
||||
d.Summary = src.Summary()
|
||||
if fs, ok := src.(FabricSource); ok {
|
||||
r := fs.Fabrics()
|
||||
|
||||
@@ -24,7 +24,10 @@ type FleetCharts struct {
|
||||
Queue TimeSeries
|
||||
KV TimeSeries
|
||||
Ready TimeSeries
|
||||
Window string
|
||||
// GPUBusy and GPUMemory are the fleet's cards by class, for a fleet that
|
||||
// has them.
|
||||
GPUBusy, GPUMemory TimeSeries
|
||||
Window string
|
||||
// Err is set when the metrics store could not be read. Shown as a line of
|
||||
// text where the charts would be: an empty panel reads as a healthy cluster
|
||||
// with nothing happening, which is the opposite of not knowing.
|
||||
@@ -90,3 +93,25 @@ func LoadFleetCharts(ctx context.Context, h History, window time.Duration,
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// LoadGPU adds the fleet's GPUs, by class: how busy on average, and how full
|
||||
// their memory. Asked only of a fleet with GPUs. A failure leaves these two
|
||||
// empty and the rest of the page as it was; they are not what the other
|
||||
// panels' all-or-nothing is about.
|
||||
func (f *FleetCharts) LoadGPU(ctx context.Context, h History, window time.Duration) {
|
||||
if f == nil || h == nil || f.Err != "" {
|
||||
return
|
||||
}
|
||||
if window <= 0 {
|
||||
window = time.Hour
|
||||
}
|
||||
step := window / 60
|
||||
if s, err := h.RangeQuery(ctx, `avg by (class) (llmgw_gpu_utilization_percent) / 100`, "class", window, step); err == nil {
|
||||
f.GPUBusy = LineChart(s, "%", 6)
|
||||
}
|
||||
if s, err := h.RangeQuery(ctx,
|
||||
`sum by (class) (llmgw_gpu_memory_used_bytes) / sum by (class) (llmgw_gpu_memory_total_bytes)`,
|
||||
"class", window, step); err == nil {
|
||||
f.GPUMemory = LineChart(s, "%", 6)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// modelEfficiency is what a model gets out of the GPUs it holds, over the
|
||||
// last quarter hour.
|
||||
type modelEfficiency struct {
|
||||
Model string
|
||||
// TokensPerSec is output tokens a second; GPUs the cards it holds now.
|
||||
TokensPerSec float64
|
||||
GPUs float64
|
||||
// Watts is what those cards draw; Util their average utilisation.
|
||||
Watts, Util float64
|
||||
// PerGPU and PerWh are the two ways of saying how much work the cards
|
||||
// do: output tokens a second per card, and per watt-hour drawn.
|
||||
PerGPU, PerWh float64
|
||||
}
|
||||
|
||||
// The queries behind the table. Each model's cards are found by joining the
|
||||
// GPU series to llmgw_gpu_assigned on the card, which carries the model.
|
||||
var efficiencyQueries = []struct {
|
||||
expr string
|
||||
into func(*modelEfficiency, float64)
|
||||
}{
|
||||
{`sum by (model) (rate(llmgw_tokens_total{direction="completion"}[15m]))`,
|
||||
func(e *modelEfficiency, v float64) { e.TokensPerSec = v }},
|
||||
{`count by (model) (llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.GPUs = v }},
|
||||
{`sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.Watts = v }},
|
||||
{`avg by (model) (llmgw_gpu_utilization_percent * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.Util = v }},
|
||||
}
|
||||
|
||||
// loadEfficiency is the table, best per card first; nil when the metrics
|
||||
// store cannot be read or no model holds a GPU.
|
||||
func loadEfficiency(ctx context.Context, h History) []modelEfficiency {
|
||||
if h == nil {
|
||||
return nil
|
||||
}
|
||||
byModel := map[string]*modelEfficiency{}
|
||||
for _, q := range efficiencyQueries {
|
||||
// The last of a short range: what History offers, and the one point
|
||||
// the table needs.
|
||||
series, err := h.RangeQuery(ctx, q.expr, "model", 10*time.Minute, 5*time.Minute)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
for _, s := range series {
|
||||
if len(s.Points) == 0 {
|
||||
continue
|
||||
}
|
||||
e := byModel[s.Name]
|
||||
if e == nil {
|
||||
e = &modelEfficiency{Model: s.Name}
|
||||
byModel[s.Name] = e
|
||||
}
|
||||
q.into(e, s.Last())
|
||||
}
|
||||
}
|
||||
var out []modelEfficiency
|
||||
for _, e := range byModel {
|
||||
if e.GPUs == 0 {
|
||||
continue // a model on the CPU; this table is about cards
|
||||
}
|
||||
e.PerGPU = e.TokensPerSec / e.GPUs
|
||||
if e.Watts > 0 {
|
||||
e.PerWh = e.TokensPerSec * 3600 / e.Watts
|
||||
}
|
||||
out = append(out, *e)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].PerGPU != out[j].PerGPU {
|
||||
return out[i].PerGPU > out[j].PerGPU
|
||||
}
|
||||
return out[i].Model < out[j].Model
|
||||
})
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"html/template"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// The GPU map is every card in the fleet, a square each, a row per node,
|
||||
// coloured by one figure: how busy, how hot, or how full its memory is. A
|
||||
// hundred cards in a table are a hundred rows to read; on the map the one
|
||||
// that is not like the others is seen before anything is read.
|
||||
|
||||
// gpuMapModes are what the map can be coloured by.
|
||||
var gpuMapModes = []struct{ Key, Label string }{
|
||||
{"util", "busy"}, {"memory", "memory"}, {"temp", "temperature"},
|
||||
}
|
||||
|
||||
type gpuMap struct {
|
||||
By string
|
||||
Modes []struct{ Key, Label string }
|
||||
Rows []gpuMapRow
|
||||
// Legend says what the colours mean in this mode.
|
||||
Legend string
|
||||
}
|
||||
|
||||
type gpuMapRow struct {
|
||||
Node string
|
||||
Cells []gpuCell
|
||||
}
|
||||
|
||||
type gpuCell struct {
|
||||
Index int
|
||||
// Model and Replica hold the card; empty is a free card.
|
||||
Model, Replica string
|
||||
// Style is the cell's colour, built here from numbers alone, which is
|
||||
// what makes it safe to put in a style attribute; Title says everything
|
||||
// known about the card.
|
||||
Style template.CSS
|
||||
Title string
|
||||
// Label is the figure written in the cell.
|
||||
Label string
|
||||
// Faulty is a card whose wear needs seeing to.
|
||||
Faulty bool
|
||||
}
|
||||
|
||||
// idleHolding is a replica holding GPUs that are doing nothing.
|
||||
type idleHolding struct {
|
||||
Model, Replica, Node string
|
||||
GPUs []int
|
||||
// MaxUtil is the busiest of its cards, now.
|
||||
MaxUtil float64
|
||||
// Idle is how long since a request was routed to it, or "" when not
|
||||
// known.
|
||||
Idle string
|
||||
}
|
||||
|
||||
// buildGPUMap draws the map from what the nodes last reported and where
|
||||
// replicas are. Nil for a fleet without GPUs.
|
||||
func buildGPUMap(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, by, prefix string) *gpuMap {
|
||||
held := heldBy(reps)
|
||||
m := &gpuMap{By: by, Modes: gpuMapModes}
|
||||
if m.By == "" {
|
||||
m.By = "util"
|
||||
}
|
||||
switch m.By {
|
||||
case "temp":
|
||||
m.Legend = "Blue is cool, red is 90 °C and over."
|
||||
case "memory":
|
||||
m.Legend = "Pale is empty, dark is full. Engines reserve most of a card's memory when they start."
|
||||
default:
|
||||
m.By = "util"
|
||||
m.Legend = "Pale is idle, dark green is fully busy. A dashed border is a card no replica holds."
|
||||
}
|
||||
for _, n := range nodes {
|
||||
if n.GPUTotal == 0 {
|
||||
continue
|
||||
}
|
||||
row := gpuMapRow{Node: strings.TrimPrefix(n.ID, prefix)}
|
||||
for i := range n.GPUTotal {
|
||||
c := gpuCell{Index: i}
|
||||
if h, ok := held[gpuKey(n.ID, i)]; ok {
|
||||
c.Model, c.Replica = h.model, h.replica
|
||||
}
|
||||
var title []string
|
||||
title = append(title, fmt.Sprintf("%s GPU %d", n.ID, i))
|
||||
if c.Model != "" {
|
||||
title = append(title, c.Model+" ("+c.Replica+")")
|
||||
} else {
|
||||
title = append(title, "free")
|
||||
}
|
||||
s, sampled := sensorOf(n, i)
|
||||
if sampled {
|
||||
if s.MemoryTotalMB > 0 && s.UtilPercent >= 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f%% busy", s.UtilPercent))
|
||||
}
|
||||
if s.MemoryTotalMB > 0 {
|
||||
title = append(title, fmt.Sprintf("%.1f of %.0f GB", s.MemoryUsedMB/1024, s.MemoryTotalMB/1024))
|
||||
}
|
||||
if s.TempC > 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f °C", s.TempC))
|
||||
}
|
||||
if s.PowerW > 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f W", s.PowerW))
|
||||
}
|
||||
if len(s.Throttle) > 0 {
|
||||
title = append(title, "held back by "+strings.Join(s.Throttle, ", "))
|
||||
}
|
||||
title = append(title, s.Problems...)
|
||||
}
|
||||
c.Title = strings.Join(title, " · ")
|
||||
style, label := cellColour(m.By, s, sampled)
|
||||
if c.Model == "" {
|
||||
style += ";border-style:dashed"
|
||||
}
|
||||
// A card wearing out is outlined whatever the map is coloured
|
||||
// by: it is the one thing on the map that needs doing.
|
||||
if len(s.Problems) > 0 {
|
||||
style += ";border:2px solid #c0392b"
|
||||
c.Faulty = true
|
||||
}
|
||||
c.Style, c.Label = template.CSS(style), label
|
||||
row.Cells = append(row.Cells, c)
|
||||
}
|
||||
m.Rows = append(m.Rows, row)
|
||||
}
|
||||
if len(m.Rows) == 0 {
|
||||
return nil
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// cellColour is a cell's background, and the figure written in it.
|
||||
func cellColour(by string, s sensorView, sampled bool) (string, string) {
|
||||
const unknown = "background:var(--panel-2,#eee)"
|
||||
switch by {
|
||||
case "temp":
|
||||
if !sampled || s.TempC <= 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
// 30 °C blue to 90 °C red.
|
||||
f := clamp01((s.TempC - 30) / 60)
|
||||
return fmt.Sprintf("background:hsl(%.0f,70%%,55%%)", 220-220*f), fmt.Sprintf("%.0f", s.TempC)
|
||||
case "memory":
|
||||
if !sampled || s.MemoryTotalMB <= 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
f := clamp01(s.MemoryUsedMB / s.MemoryTotalMB)
|
||||
return fmt.Sprintf("background:hsl(265,45%%,%.0f%%)", 90-55*f), fmt.Sprintf("%.0f", 100*f)
|
||||
}
|
||||
if !sampled || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
f := clamp01(s.UtilPercent / 100)
|
||||
return fmt.Sprintf("background:hsl(140,%.0f%%,%.0f%%)", 15+50*f, 88-50*f), fmt.Sprintf("%.0f", s.UtilPercent)
|
||||
}
|
||||
|
||||
// idleHoldings are the replicas whose GPUs all read under 5% busy now:
|
||||
// capacity held and not asked for, the first place to find room.
|
||||
func idleHoldings(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, prefix string) []idleHolding {
|
||||
byID := map[string]adminapi.NodeView{}
|
||||
for _, n := range nodes {
|
||||
byID[n.ID] = n
|
||||
}
|
||||
var out []idleHolding
|
||||
for _, r := range reps {
|
||||
if r.State != "ready" {
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
if len(sh.GPUs) == 0 {
|
||||
continue
|
||||
}
|
||||
h := idleHolding{Model: r.Model, Replica: r.ID, Node: strings.TrimPrefix(sh.Node, prefix), GPUs: sh.GPUs}
|
||||
idle, known := true, true
|
||||
for _, g := range sh.GPUs {
|
||||
s, ok := sensorOf(byID[sh.Node], g)
|
||||
if !ok || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
|
||||
known = false
|
||||
break
|
||||
}
|
||||
h.MaxUtil = max(h.MaxUtil, s.UtilPercent)
|
||||
if s.UtilPercent >= 5 {
|
||||
idle = false
|
||||
}
|
||||
}
|
||||
if !known || !idle {
|
||||
continue
|
||||
}
|
||||
if r.IdleSec >= 0 {
|
||||
h.Idle = (time.Duration(r.IdleSec) * time.Second).String()
|
||||
}
|
||||
out = append(out, h)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return len(out[i].GPUs) > len(out[j].GPUs) })
|
||||
return out
|
||||
}
|
||||
|
||||
type holder struct{ model, replica string }
|
||||
|
||||
func heldBy(reps []adminapi.ReplicaView) map[string]holder {
|
||||
out := map[string]holder{}
|
||||
for _, r := range reps {
|
||||
switch r.State {
|
||||
case "stopped", "failed":
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
for _, g := range sh.GPUs {
|
||||
out[gpuKey(sh.Node, g)] = holder{r.Model, r.ID}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func gpuKey(node string, gpu int) string { return fmt.Sprintf("%s/%d", node, gpu) }
|
||||
|
||||
// sensorView is a card's last sample.
|
||||
type sensorView = struct {
|
||||
TempC, PowerW, UtilPercent, MemoryUsedMB, MemoryTotalMB float64
|
||||
Throttle []string
|
||||
Problems []string
|
||||
}
|
||||
|
||||
func sensorOf(n adminapi.NodeView, gpu int) (sensorView, bool) {
|
||||
if n.Load == nil {
|
||||
return sensorView{}, false
|
||||
}
|
||||
for _, s := range n.Load.GPUs {
|
||||
if s.Index == gpu {
|
||||
return sensorView{TempC: s.TempC, PowerW: s.PowerW, UtilPercent: s.UtilPercent,
|
||||
MemoryUsedMB: s.MemoryUsedMB, MemoryTotalMB: s.MemoryTotalMB, Throttle: s.Throttle,
|
||||
Problems: s.Wear.Problems()}, true
|
||||
}
|
||||
}
|
||||
return sensorView{}, false
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
)
|
||||
|
||||
// mapFleet is a node of four cards: chat holds two and is idle, code holds
|
||||
// one and is busy, one is free.
|
||||
type mapFleet struct{ fakeSource }
|
||||
|
||||
func (mapFleet) Nodes() []adminapi.NodeView {
|
||||
busy := func(i int, util, temp float64) inventory.GPUSensor {
|
||||
return inventory.GPUSensor{Index: i, UtilPercent: util, TempC: temp, MemoryUsedMB: 40000, MemoryTotalMB: 81559, PowerW: 300}
|
||||
}
|
||||
return []adminapi.NodeView{{ID: "gpu07", State: "ready", GPUTotal: 4, GPUFree: 1, Classes: []string{"h100-80g"},
|
||||
Load: &adminapi.LoadView{GPUs: []inventory.GPUSensor{busy(0, 1, 35), busy(1, 0, 34), busy(2, 93, 81), worn(busy(3, 0, 30))}}}}
|
||||
}
|
||||
|
||||
func worn(s inventory.GPUSensor) inventory.GPUSensor {
|
||||
s.Wear = &inventory.GPUWear{RemapPending: true, XIDs: 1, LastXID: 79, LastXIDText: "GPU has fallen off the bus."}
|
||||
return s
|
||||
}
|
||||
|
||||
func (mapFleet) Replicas(string) []adminapi.ReplicaView {
|
||||
return []adminapi.ReplicaView{
|
||||
{ID: "chat-1", Model: "chat", State: "ready", IdleSec: 1800, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{0, 1}}}},
|
||||
{ID: "code-1", Model: "code", State: "ready", IdleSec: 2, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{2}}}},
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClusterPageMapsEveryCardAndNamesTheIdleOnes(t *testing.T) {
|
||||
_, body := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if n := strings.Count(body, `class="gpu-cell"`); n != 4 {
|
||||
t.Errorf("%d cards on the map, want 4", n)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"gpu07 GPU 2 · code (code-1) · 93% busy",
|
||||
"gpu07 GPU 3 · free",
|
||||
"memory rows remapped, waiting for a reset · XID 79 (GPU has fallen off the bus.)",
|
||||
"border:2px solid #c0392b",
|
||||
"border-style:dashed",
|
||||
"Cards held and doing nothing",
|
||||
"<code>chat-1</code></td><td><code>gpu07</code>", "30m0s ago",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "<code>code-1</code></td><td><code>gpu07</code>") {
|
||||
t.Error("a busy replica was listed as idle")
|
||||
}
|
||||
_, temp := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/?gpumap=temp", nil)
|
||||
if !strings.Contains(temp, "<strong>temperature</strong>") || !strings.Contains(temp, ">81</span>") {
|
||||
t.Error("the map coloured by temperature does not say so, or show it")
|
||||
}
|
||||
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(cpu, "gpu-map") {
|
||||
t.Error("a fleet without GPUs has a GPU map")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachModelSaysWhatItGetsFromItsCards(t *testing.T) {
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{}}
|
||||
for i, vals := range [][2]float64{{120, 30}, {2, 2}, {1200, 400}, {80, 40}} {
|
||||
h.byExpr[efficiencyQueries[i].expr] = []promq.Series{ts("chat", vals[0], vals[0]), ts("code", vals[1], vals[1])}
|
||||
}
|
||||
got := loadEfficiency(context.Background(), h)
|
||||
if len(got) != 2 || got[0].Model != "chat" {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
c := got[0]
|
||||
if c.PerGPU != 60 || c.PerWh != 120*3600/1200.0 || c.Util != 80 {
|
||||
t.Errorf("chat: %+v", c)
|
||||
}
|
||||
if got[1].PerGPU != 15 {
|
||||
t.Errorf("code: %+v", got[1])
|
||||
}
|
||||
}
|
||||
@@ -25,7 +25,8 @@ func TestEveryPageDrawsTheSameMenu(t *testing.T) {
|
||||
p.RegisterAdminMirror(mux, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{}}})
|
||||
|
||||
pages := map[string]string{
|
||||
"/admin/users": "People", "/admin/quotas": "Quotas", "/admin/mirror": "Mirror",
|
||||
// Limits and usage are tabs of the people page, which they mark.
|
||||
"/admin/users": "People", "/admin/quotas": "People", "/admin/mirror": "Mirror",
|
||||
}
|
||||
var first string
|
||||
for path, current := range pages {
|
||||
@@ -44,7 +45,7 @@ func TestEveryPageDrawsTheSameMenu(t *testing.T) {
|
||||
t.Errorf("%s draws a different menu:\n%s\nfrom:\n%s", path, plain, first)
|
||||
}
|
||||
}
|
||||
for _, link := range []string{"/admin", "/admin/users", "/admin/quotas", "/admin/usage",
|
||||
for _, link := range []string{"/admin", "/admin/users",
|
||||
"/admin/mirror", "/chat", "/profile", "/logout"} {
|
||||
if !strings.Contains(first, `"`+link+`"`) {
|
||||
t.Errorf("the menu lacks %s", link)
|
||||
|
||||
@@ -40,8 +40,10 @@ type nodeEngine struct {
|
||||
// nodeCharts are the node's recent history, from the metrics store.
|
||||
type nodeCharts struct {
|
||||
CPU, Memory, Load, EngineCPU, EngineMemory, Queue TimeSeries
|
||||
Window string
|
||||
Err string
|
||||
// The GPUs', one line per card; empty on a node without them.
|
||||
GPUUtil, GPUMemory, GPUTemp, GPUPower TimeSeries
|
||||
Window string
|
||||
Err string
|
||||
}
|
||||
|
||||
// nodePanels are the charts, each a query scoped to this node. %s is the
|
||||
@@ -63,6 +65,23 @@ var nodePanels = []struct {
|
||||
func(c *nodeCharts, t TimeSeries) { c.EngineMemory = t }},
|
||||
}
|
||||
|
||||
// gpuPanels are the GPUs' charts, one line per card, for a node that has
|
||||
// them. Read from the heartbeat's samples, not from what the node said when
|
||||
// it registered.
|
||||
var gpuPanels = []struct {
|
||||
expr, unit string
|
||||
into func(*nodeCharts, TimeSeries)
|
||||
}{
|
||||
{`llmgw_gpu_utilization_percent{node=%s} / 100`, "%",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUUtil = t }},
|
||||
{`llmgw_gpu_memory_used_bytes{node=%s}`, "bytes",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUMemory = t }},
|
||||
{`llmgw_gpu_temperature_celsius{node=%s}`, "",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUTemp = t }},
|
||||
{`llmgw_gpu_power_watts{node=%s}`, "",
|
||||
func(c *nodeCharts, t TimeSeries) { c.GPUPower = t }},
|
||||
}
|
||||
|
||||
// NodePage shows one node. Its charts come from the metrics store when one
|
||||
// is configured; the current figures come from the node's last heartbeat
|
||||
// either way.
|
||||
@@ -115,14 +134,14 @@ func NodePage(src Source, history History, window time.Duration, onAdminListener
|
||||
sort.Slice(d.Engines, func(i, j int) bool { return d.Engines[i].CPUCores > d.Engines[j].CPUCores })
|
||||
|
||||
if history != nil {
|
||||
d.Charts = loadNodeCharts(r.Context(), history, id, replicas, window, log)
|
||||
d.Charts = loadNodeCharts(r.Context(), history, id, replicas, d.Node.GPUTotal, window, log)
|
||||
}
|
||||
noStore(w)
|
||||
render(w, "node", d)
|
||||
}
|
||||
}
|
||||
|
||||
func loadNodeCharts(ctx context.Context, h History, node string, replicas []string,
|
||||
func loadNodeCharts(ctx context.Context, h History, node string, replicas []string, gpus int,
|
||||
window time.Duration, log *slog.Logger,
|
||||
) *nodeCharts {
|
||||
step := window / 60
|
||||
@@ -139,6 +158,19 @@ func loadNodeCharts(ctx context.Context, h History, node string, replicas []stri
|
||||
}
|
||||
p.into(out, LineChart(series, p.unit, 4))
|
||||
}
|
||||
// Every card its own line, up to the eight a node commonly has.
|
||||
if gpus > 0 {
|
||||
for _, p := range gpuPanels {
|
||||
series, err := h.RangeQuery(ctx, fmt.Sprintf(p.expr, quoted), "gpu", window, step)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for i := range series {
|
||||
series[i].Name = "GPU " + series[i].Name
|
||||
}
|
||||
p.into(out, LineChart(series, p.unit, max(gpus, 8)))
|
||||
}
|
||||
}
|
||||
// The engines' queues, for the replicas that are here now.
|
||||
if len(replicas) > 0 {
|
||||
re := make([]string, len(replicas))
|
||||
|
||||
@@ -205,3 +205,68 @@ func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// gpuSource is a node with two GPUs, one busy and one idle.
|
||||
type gpuSource struct{ fakeSource }
|
||||
|
||||
func (gpuSource) Nodes() []adminapi.NodeView {
|
||||
n := fakeSource{}.Nodes()[0]
|
||||
n.GPUTotal = 2
|
||||
n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
|
||||
GPUs: []inventory.GPUSensor{
|
||||
{Index: 0, TempC: 71, PowerW: 612, PowerLimitW: 700, UtilPercent: 97, MemUtilPercent: 64,
|
||||
MemoryUsedMB: 72909, MemoryTotalMB: 81559, SMClockMHz: 1980, ECCUncorrected: 0},
|
||||
{Index: 1, TempC: 30, PowerW: 70, PowerLimitW: 700, UtilPercent: 0, MemUtilPercent: 0,
|
||||
MemoryUsedMB: 512, MemoryTotalMB: 81559, SMClockMHz: 345, ECCUncorrected: 3},
|
||||
}}
|
||||
return []adminapi.NodeView{n}
|
||||
}
|
||||
|
||||
func TestAGPUNodesPageShowsHowBusyEachCardIsAndOverTime(t *testing.T) {
|
||||
q := func(expr string) string { return strings.Replace(expr, "%s", `"llmgw-node01"`, 1) }
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{
|
||||
q(`llmgw_gpu_utilization_percent{node=%s} / 100`): {ts("0", 0.9, 0.97), ts("1", 0, 0)},
|
||||
q(`llmgw_gpu_memory_used_bytes{node=%s}`): {ts("0", 7e10, 7.6e10)},
|
||||
q(`llmgw_gpu_temperature_celsius{node=%s}`): {ts("0", 65, 71)},
|
||||
q(`llmgw_gpu_power_watts{node=%s}`): {ts("0", 500, 612)},
|
||||
}}
|
||||
_, body := get(t, nodeMux(gpuSource{}, h), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, want := range []string{
|
||||
"97%", "memory 64%", "71.2 <span class=\"note\">of 80 GB", "1980 MHz",
|
||||
"3 uncorrected ECC errors",
|
||||
"GPU busy, by card", "GPU memory used, by card", "GPU temperature, °C", "GPU power, W", "GPU 0",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// A node without GPUs is not asked about them.
|
||||
h2 := &fakeHistory{}
|
||||
get(t, nodeMux(cpuFleet{}, h2), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, e := range h2.asked {
|
||||
if strings.Contains(e, "llmgw_gpu_") {
|
||||
t.Errorf("a node with no GPU was asked for %s", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClusterPageDrawsItsGPUsByClass(t *testing.T) {
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{
|
||||
`avg by (class) (llmgw_gpu_utilization_percent) / 100`: {ts("h100-80g", 0.2, 0.6)},
|
||||
`sum by (class) (llmgw_gpu_memory_used_bytes) / sum by (class) (llmgw_gpu_memory_total_bytes)`: {ts("h100-80g", 0.8, 0.9)},
|
||||
}}
|
||||
_, body := get(t, Dashboard(fakeSource{}, nil, "v1", time.Second, true, NavExtras{History: h, Window: time.Hour}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if !strings.Contains(body, "GPUs busy, by class") || !strings.Contains(body, "GPU memory in use, by class") {
|
||||
t.Error("a fleet with GPUs has no GPU charts")
|
||||
}
|
||||
h2 := &fakeHistory{}
|
||||
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{History: h2, Window: time.Hour}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(cpu, "GPUs busy") {
|
||||
t.Error("a fleet without GPUs was drawn GPU charts")
|
||||
}
|
||||
for _, e := range h2.asked {
|
||||
if strings.Contains(e, "llmgw_gpu_") {
|
||||
t.Errorf("a fleet without GPUs was asked for %s", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
@@ -70,6 +71,9 @@ type orgPageData struct {
|
||||
func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced Priced) {
|
||||
p.orgs = s
|
||||
mux.HandleFunc("GET /org", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
|
||||
if p.toPeople(w, r, user, "") {
|
||||
return
|
||||
}
|
||||
orgs, err := p.viewableOrgs(r.Context(), s, user)
|
||||
if err == nil && len(orgs) == 0 && user.IsAdmin {
|
||||
// Said, not a 404: an administrator following the link on a
|
||||
@@ -89,6 +93,9 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
|
||||
}))
|
||||
mux.HandleFunc("GET /org/{id}", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
|
||||
id := r.PathValue("id")
|
||||
if p.toPeople(w, r, user, id) {
|
||||
return
|
||||
}
|
||||
if !user.IsAdmin {
|
||||
ok, err := s.ManagesOrg(r.Context(), user.ID, id)
|
||||
if err != nil || !ok {
|
||||
@@ -102,6 +109,22 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
|
||||
}))
|
||||
}
|
||||
|
||||
// toPeople sends an administrator to the people page, which shows an
|
||||
// organization's members, limits and use beside the whole tree and can change
|
||||
// them; this page stays for the managers who may not see that one. It says
|
||||
// whether it did.
|
||||
func (p *AccountPages) toPeople(w http.ResponseWriter, r *http.Request, user store.User, orgID string) bool {
|
||||
if !user.IsAdmin || p.peopleStore == nil {
|
||||
return false
|
||||
}
|
||||
to := "/admin/users"
|
||||
if orgID != "" {
|
||||
to += "?sel=org:" + url.QueryEscape(orgID)
|
||||
}
|
||||
http.Redirect(w, r, to, http.StatusSeeOther)
|
||||
return true
|
||||
}
|
||||
|
||||
// viewableOrgs is every organization an administrator may look at, or the
|
||||
// ones a person manages.
|
||||
func (p *AccountPages) viewableOrgs(ctx context.Context, s OrgViewer, user store.User) ([]store.Org, error) {
|
||||
|
||||
@@ -124,3 +124,22 @@ func TestWithNoOrganizationsAnAdministratorIsToldSo(t *testing.T) {
|
||||
t.Error("the menu links to organizations when there are none")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdministratorIsSentToThePeoplePageForAnOrganization(t *testing.T) {
|
||||
// Its members, limits and use are a panel there, beside the whole tree
|
||||
// and changeable; /org stays for the managers who cannot see that page.
|
||||
p := NewAccountPages(fakeAccounts{user: store.User{ID: "a", IsAdmin: true}}, nil, nil, "v0",
|
||||
slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsers(mux, &fakeAdmin{})
|
||||
p.RegisterOrgPages(mux, fakeOrgs{}, nil)
|
||||
for path, to := range map[string]string{"/org": "/admin/users", "/org/org-s": "/admin/users?sel=org:org-s"} {
|
||||
w := mirrorGet(t, mux, path)
|
||||
if w.Code != http.StatusSeeOther || w.Header().Get("Location") != to {
|
||||
t.Errorf("%s: %d to %q, want %s", path, w.Code, w.Header().Get("Location"), to)
|
||||
}
|
||||
}
|
||||
if menu := menuRe.FindString(mirrorGet(t, mux, "/admin/users").Body.String()); strings.Contains(menu, `href="/org"`) {
|
||||
t.Error("the administrator's menu still offers /org beside People")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// signupAccounts is an account service whose sign-up answers with err.
|
||||
type signupAccounts struct {
|
||||
fakeAccounts
|
||||
err error
|
||||
}
|
||||
|
||||
func (s signupAccounts) SignUp(context.Context, string, string, string) error { return s.err }
|
||||
func (signupAccounts) SignupOpen() bool { return true }
|
||||
|
||||
func signupBody(t *testing.T, err error) string {
|
||||
t.Helper()
|
||||
mux := http.NewServeMux()
|
||||
NewAccountPages(signupAccounts{err: err}, nil, nil, "v0", slog.New(slog.DiscardHandler)).Register(mux)
|
||||
form := url.Values{"email": {"a@b.c"}, "name": {"A"}, "password": {"long enough password"}}
|
||||
req := httptest.NewRequest(http.MethodPost, "/signup", strings.NewReader(form.Encode()))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec.Body.String()
|
||||
}
|
||||
|
||||
func TestASignupThatFailsSaysSoWithAReferenceAndNoDatabaseWords(t *testing.T) {
|
||||
body := signupBody(t, errors.New(`duplicate key value violates unique constraint "users_email_key"`))
|
||||
if strings.Contains(body, "users_email_key") || strings.Contains(body, "constraint") {
|
||||
t.Errorf("the database's own words reached the page: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "was not created") || !strings.Contains(body, "reference: err-") {
|
||||
t.Errorf("no plain failure with a reference to give an operator: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASignupSaysWhatToDoIfTheAddressHadAnAccount(t *testing.T) {
|
||||
// The service answers a taken address as it does a new one, so nobody
|
||||
// can find out from this page who has an account; what the page can do
|
||||
// is say what to do in either case.
|
||||
body := signupBody(t, nil)
|
||||
if !strings.Contains(body, "already had an account") || !strings.Contains(body, `href="/login"`) ||
|
||||
!strings.Contains(body, "reset it") {
|
||||
t.Errorf("the done page leaves someone who already had an account stuck: %s", body)
|
||||
}
|
||||
}
|
||||
Loaded 100 of 112 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user