Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
febcf251e4 | ||
|
|
fffed716be | ||
|
|
1b89d5cdea | ||
|
|
477d0bf19d | ||
|
|
372ef64889 | ||
|
|
82a63cfc6e | ||
|
|
ff554c7062 | ||
|
|
b6a802c03b |
No files matched your search
@@ -48,6 +48,78 @@ and a figure the driver does not give is left out rather than shown as zero.
|
||||
limit, clock, and what holds cards back.
|
||||
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
|
||||
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
|
||||
- **The cluster page's GPU map**: every card a square, a row per node,
|
||||
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
|
||||
with what holds it and its figures under the pointer; free cards dashed.
|
||||
- **Cards held and doing nothing**: the replicas whose every card reads under
|
||||
5% busy, with when they were last asked — the first place to find room.
|
||||
- **What each model gets from its cards**, on the cluster page and in
|
||||
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
|
||||
draw and average utilisation, over the last quarter hour.
|
||||
- **What wears a card out**, read every minute: XID errors from the kernel's
|
||||
log (counted from when the agent started), memory rows remapped or pages
|
||||
retired and whether either waits for a reset, a card out of spare rows,
|
||||
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
|
||||
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
|
||||
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
|
||||
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
|
||||
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
|
||||
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
|
||||
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
|
||||
`GPUNVLinkErrorsClimbing`.
|
||||
- **GPU time by owner** on the usage page: the cards each owner's models
|
||||
held over the window, charted and summed into GPU-hours, whether or not the
|
||||
models were asked anything — the time a GPU quota is about. From the
|
||||
metrics store, as far back as it keeps history.
|
||||
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
|
||||
each card and whose quota it counts against, for joining the GPU series to
|
||||
models and tenants.
|
||||
|
||||
### People, limits and usage are one page
|
||||
|
||||
Managing a team meant its members on the people page, its quota on the quotas
|
||||
page and its spend on the usage page, each linking to the others. They are now
|
||||
one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
|
||||
each at the address it had before, so links and bookmarks still land. The menu
|
||||
has one entry for them, People.
|
||||
|
||||
Choosing an organization, team, project or person in the tree, or a person in
|
||||
the account list, opens a panel beside it with everything about that one
|
||||
thing: its own limits and what it is held to from above, spend against its
|
||||
budget, its per-member limit, its capacity quota, and its requests, tokens
|
||||
and spend over the last week, drawn. Its limits can be set or changed there,
|
||||
and the answer comes back to the same panel.
|
||||
|
||||
The administrator's menu no longer has an Organizations entry beside it: `/org`
|
||||
and `/org/<id>` send an administrator to that organization's panel here. The
|
||||
page stays as it was for an organization's managers, who cannot see this one.
|
||||
|
||||
### The chat page says whether a GPU or a CPU answered
|
||||
|
||||
A model can have replicas on a card and on CPUs at once, and the two answer
|
||||
at very different speeds; the chat page gave no way to tell which one a reply
|
||||
came from. The picker now says where each model runs — `GPU`, `CPU` or
|
||||
`GPU + CPU` — and the line under each reply names what answered it: `GPU
|
||||
6000-24g` or `CPU`, beside its tokens a second.
|
||||
|
||||
Every answer through the gateway carries the same as `x-llmgw-hardware`:
|
||||
`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
|
||||
none.
|
||||
|
||||
### A card shared with something else is judged every heartbeat
|
||||
|
||||
A card with another process on it when its agent started was kept out of
|
||||
placement, and stayed out — "5014 MiB already allocated by another process" —
|
||||
after that process let go of it, until the agent was restarted. Found on a
|
||||
node shared with ComfyUI. The other way round, a tenant arriving after
|
||||
registration was never seen, and a card the agent's own engine held when the
|
||||
agent restarted read as someone else's.
|
||||
|
||||
The agent now reads every card's memory per process on each heartbeat,
|
||||
leaves out what its own engines hold, and sends the rest; the control plane
|
||||
keeps a card out of placement on that reading, and lets it back in when it
|
||||
falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
|
||||
agent too old to send it keeps the reading from registration.
|
||||
|
||||
### A GPU placement no longer runs on the CPU unnoticed
|
||||
|
||||
|
||||
@@ -1561,8 +1561,27 @@ type GPUSensor struct {
|
||||
SmClockMhz float64 `protobuf:"fixed64,11,opt,name=sm_clock_mhz,json=smClockMhz,proto3" json:"sm_clock_mhz,omitempty"`
|
||||
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
|
||||
EccUncorrected float64 `protobuf:"fixed64,12,opt,name=ecc_uncorrected,json=eccUncorrected,proto3" json:"ecc_uncorrected,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
// What wears the card out, read every minute; has_wear says these are
|
||||
// present. Counts of -1 are not reported for this card.
|
||||
HasWear bool `protobuf:"varint,13,opt,name=has_wear,json=hasWear,proto3" json:"has_wear,omitempty"`
|
||||
PcieReplays int64 `protobuf:"varint,14,opt,name=pcie_replays,json=pcieReplays,proto3" json:"pcie_replays,omitempty"`
|
||||
RetiredPagesSbe int64 `protobuf:"varint,15,opt,name=retired_pages_sbe,json=retiredPagesSbe,proto3" json:"retired_pages_sbe,omitempty"`
|
||||
RetiredPagesDbe int64 `protobuf:"varint,16,opt,name=retired_pages_dbe,json=retiredPagesDbe,proto3" json:"retired_pages_dbe,omitempty"`
|
||||
RetirementPending bool `protobuf:"varint,17,opt,name=retirement_pending,json=retirementPending,proto3" json:"retirement_pending,omitempty"`
|
||||
RemappedRowsCorrectable int64 `protobuf:"varint,18,opt,name=remapped_rows_correctable,json=remappedRowsCorrectable,proto3" json:"remapped_rows_correctable,omitempty"`
|
||||
RemappedRowsUncorrectable int64 `protobuf:"varint,19,opt,name=remapped_rows_uncorrectable,json=remappedRowsUncorrectable,proto3" json:"remapped_rows_uncorrectable,omitempty"`
|
||||
RemapPending bool `protobuf:"varint,20,opt,name=remap_pending,json=remapPending,proto3" json:"remap_pending,omitempty"`
|
||||
RemapFailure bool `protobuf:"varint,21,opt,name=remap_failure,json=remapFailure,proto3" json:"remap_failure,omitempty"`
|
||||
NvlinkCrcErrors int64 `protobuf:"varint,22,opt,name=nvlink_crc_errors,json=nvlinkCrcErrors,proto3" json:"nvlink_crc_errors,omitempty"`
|
||||
NvlinkReplayErrors int64 `protobuf:"varint,23,opt,name=nvlink_replay_errors,json=nvlinkReplayErrors,proto3" json:"nvlink_replay_errors,omitempty"`
|
||||
NvlinkRecoveryErrors int64 `protobuf:"varint,24,opt,name=nvlink_recovery_errors,json=nvlinkRecoveryErrors,proto3" json:"nvlink_recovery_errors,omitempty"`
|
||||
// xid_errors are the driver's error events since the agent started.
|
||||
XidErrors int64 `protobuf:"varint,25,opt,name=xid_errors,json=xidErrors,proto3" json:"xid_errors,omitempty"`
|
||||
LastXid int32 `protobuf:"varint,26,opt,name=last_xid,json=lastXid,proto3" json:"last_xid,omitempty"`
|
||||
LastXidText string `protobuf:"bytes,27,opt,name=last_xid_text,json=lastXidText,proto3" json:"last_xid_text,omitempty"`
|
||||
LastXidUnix int64 `protobuf:"varint,28,opt,name=last_xid_unix,json=lastXidUnix,proto3" json:"last_xid_unix,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *GPUSensor) Reset() {
|
||||
@@ -1679,6 +1698,118 @@ func (x *GPUSensor) GetEccUncorrected() float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetHasWear() bool {
|
||||
if x != nil {
|
||||
return x.HasWear
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetPcieReplays() int64 {
|
||||
if x != nil {
|
||||
return x.PcieReplays
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetiredPagesSbe() int64 {
|
||||
if x != nil {
|
||||
return x.RetiredPagesSbe
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetiredPagesDbe() int64 {
|
||||
if x != nil {
|
||||
return x.RetiredPagesDbe
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRetirementPending() bool {
|
||||
if x != nil {
|
||||
return x.RetirementPending
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemappedRowsCorrectable() int64 {
|
||||
if x != nil {
|
||||
return x.RemappedRowsCorrectable
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemappedRowsUncorrectable() int64 {
|
||||
if x != nil {
|
||||
return x.RemappedRowsUncorrectable
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemapPending() bool {
|
||||
if x != nil {
|
||||
return x.RemapPending
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetRemapFailure() bool {
|
||||
if x != nil {
|
||||
return x.RemapFailure
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkCrcErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkCrcErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkReplayErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkReplayErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetNvlinkRecoveryErrors() int64 {
|
||||
if x != nil {
|
||||
return x.NvlinkRecoveryErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetXidErrors() int64 {
|
||||
if x != nil {
|
||||
return x.XidErrors
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXid() int32 {
|
||||
if x != nil {
|
||||
return x.LastXid
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXidText() string {
|
||||
if x != nil {
|
||||
return x.LastXidText
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetLastXidUnix() int64 {
|
||||
if x != nil {
|
||||
return x.LastXidUnix
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
type ShardLoad struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -2205,9 +2336,17 @@ type Heartbeat struct {
|
||||
GpuFabric []*GPUFabricState `protobuf:"bytes,16,rep,name=gpu_fabric,json=gpuFabric,proto3" json:"gpu_fabric,omitempty"`
|
||||
Imex *IMEX `protobuf:"bytes,17,opt,name=imex,proto3" json:"imex,omitempty"`
|
||||
// cpu_count as in Register, for a VM given more CPUs while it runs.
|
||||
CpuCount int32 `protobuf:"varint,18,opt,name=cpu_count,json=cpuCount,proto3" json:"cpu_count,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
CpuCount int32 `protobuf:"varint,18,opt,name=cpu_count,json=cpuCount,proto3" json:"cpu_count,omitempty"`
|
||||
// gpu_foreign_memory_mb, indexed by GPU, is what processes that are not
|
||||
// this agent's shards hold on each card, read afresh every heartbeat. What
|
||||
// keeps a card out of placement for another tenant is decided from this
|
||||
// rather than from registration: a card that had something else on it when
|
||||
// the agent started is not taken forever, and one that gains a tenant later
|
||||
// is not handed out. Empty from an agent that does not measure it, or from
|
||||
// one that could not this time.
|
||||
GpuForeignMemoryMb []int32 `protobuf:"varint,19,rep,packed,name=gpu_foreign_memory_mb,json=gpuForeignMemoryMb,proto3" json:"gpu_foreign_memory_mb,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Heartbeat) Reset() {
|
||||
@@ -2366,6 +2505,13 @@ func (x *Heartbeat) GetCpuCount() int32 {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *Heartbeat) GetGpuForeignMemoryMb() []int32 {
|
||||
if x != nil {
|
||||
return x.GpuForeignMemoryMb
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
// versions. Nothing more.
|
||||
//
|
||||
@@ -4051,7 +4197,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
" \x01(\x01R\tcpuPowerW\x12 \n" +
|
||||
"\finlet_temp_c\x18\v \x01(\x01R\n" +
|
||||
"inletTempC\x12'\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xbd\x03\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xde\b\n" +
|
||||
"\tGPUSensor\x12\x14\n" +
|
||||
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x15\n" +
|
||||
"\x06temp_c\x18\x02 \x01(\x01R\x05tempC\x12\"\n" +
|
||||
@@ -4066,7 +4212,24 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
" \x01(\x01R\rmemoryTotalMb\x12 \n" +
|
||||
"\fsm_clock_mhz\x18\v \x01(\x01R\n" +
|
||||
"smClockMhz\x12'\n" +
|
||||
"\x0fecc_uncorrected\x18\f \x01(\x01R\x0eeccUncorrected\"m\n" +
|
||||
"\x0fecc_uncorrected\x18\f \x01(\x01R\x0eeccUncorrected\x12\x19\n" +
|
||||
"\bhas_wear\x18\r \x01(\bR\ahasWear\x12!\n" +
|
||||
"\fpcie_replays\x18\x0e \x01(\x03R\vpcieReplays\x12*\n" +
|
||||
"\x11retired_pages_sbe\x18\x0f \x01(\x03R\x0fretiredPagesSbe\x12*\n" +
|
||||
"\x11retired_pages_dbe\x18\x10 \x01(\x03R\x0fretiredPagesDbe\x12-\n" +
|
||||
"\x12retirement_pending\x18\x11 \x01(\bR\x11retirementPending\x12:\n" +
|
||||
"\x19remapped_rows_correctable\x18\x12 \x01(\x03R\x17remappedRowsCorrectable\x12>\n" +
|
||||
"\x1bremapped_rows_uncorrectable\x18\x13 \x01(\x03R\x19remappedRowsUncorrectable\x12#\n" +
|
||||
"\rremap_pending\x18\x14 \x01(\bR\fremapPending\x12#\n" +
|
||||
"\rremap_failure\x18\x15 \x01(\bR\fremapFailure\x12*\n" +
|
||||
"\x11nvlink_crc_errors\x18\x16 \x01(\x03R\x0fnvlinkCrcErrors\x120\n" +
|
||||
"\x14nvlink_replay_errors\x18\x17 \x01(\x03R\x12nvlinkReplayErrors\x124\n" +
|
||||
"\x16nvlink_recovery_errors\x18\x18 \x01(\x03R\x14nvlinkRecoveryErrors\x12\x1d\n" +
|
||||
"\n" +
|
||||
"xid_errors\x18\x19 \x01(\x03R\txidErrors\x12\x19\n" +
|
||||
"\blast_xid\x18\x1a \x01(\x05R\alastXid\x12\"\n" +
|
||||
"\rlast_xid_text\x18\x1b \x01(\tR\vlastXidText\x12\"\n" +
|
||||
"\rlast_xid_unix\x18\x1c \x01(\x03R\vlastXidUnix\"m\n" +
|
||||
"\tShardLoad\x12\x19\n" +
|
||||
"\bshard_id\x18\x01 \x01(\tR\ashardId\x12\x1b\n" +
|
||||
"\tcpu_cores\x18\x02 \x01(\x01R\bcpuCores\x12(\n" +
|
||||
@@ -4112,7 +4275,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x06status\x18\x03 \x01(\tR\x06status\x12!\n" +
|
||||
"\fcluster_uuid\x18\x04 \x01(\tR\vclusterUuid\x12\x1b\n" +
|
||||
"\tclique_id\x18\x05 \x01(\tR\bcliqueId\x12\x1c\n" +
|
||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xf6\x06\n" +
|
||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xa9\a\n" +
|
||||
"\tHeartbeat\x12\x17\n" +
|
||||
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12+\n" +
|
||||
"\x12gpu_memory_used_mb\x18\x02 \x03(\x05R\x0fgpuMemoryUsedMb\x12!\n" +
|
||||
@@ -4133,7 +4296,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"gpu_fabric\x18\x10 \x03(\v2\x18.llmgw.v1.GPUFabricStateR\tgpuFabric\x12\"\n" +
|
||||
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12\x1b\n" +
|
||||
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x1aA\n" +
|
||||
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x121\n" +
|
||||
"\x15gpu_foreign_memory_mb\x18\x13 \x03(\x05R\x12gpuForeignMemoryMb\x1aA\n" +
|
||||
"\x13EngineVersionsEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
||||
|
||||
@@ -314,6 +314,25 @@ message GPUSensor {
|
||||
double sm_clock_mhz = 11;
|
||||
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
|
||||
double ecc_uncorrected = 12;
|
||||
// What wears the card out, read every minute; has_wear says these are
|
||||
// present. Counts of -1 are not reported for this card.
|
||||
bool has_wear = 13;
|
||||
int64 pcie_replays = 14;
|
||||
int64 retired_pages_sbe = 15;
|
||||
int64 retired_pages_dbe = 16;
|
||||
bool retirement_pending = 17;
|
||||
int64 remapped_rows_correctable = 18;
|
||||
int64 remapped_rows_uncorrectable = 19;
|
||||
bool remap_pending = 20;
|
||||
bool remap_failure = 21;
|
||||
int64 nvlink_crc_errors = 22;
|
||||
int64 nvlink_replay_errors = 23;
|
||||
int64 nvlink_recovery_errors = 24;
|
||||
// xid_errors are the driver's error events since the agent started.
|
||||
int64 xid_errors = 25;
|
||||
int32 last_xid = 26;
|
||||
string last_xid_text = 27;
|
||||
int64 last_xid_unix = 28;
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
@@ -455,6 +474,15 @@ message Heartbeat {
|
||||
IMEX imex = 17;
|
||||
// cpu_count as in Register, for a VM given more CPUs while it runs.
|
||||
int32 cpu_count = 18;
|
||||
|
||||
// gpu_foreign_memory_mb, indexed by GPU, is what processes that are not
|
||||
// this agent's shards hold on each card, read afresh every heartbeat. What
|
||||
// keeps a card out of placement for another tenant is decided from this
|
||||
// rather than from registration: a card that had something else on it when
|
||||
// the agent started is not taken forever, and one that gains a tenant later
|
||||
// is not handed out. Empty from an agent that does not measure it, or from
|
||||
// one that could not this time.
|
||||
repeated int32 gpu_foreign_memory_mb = 19;
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
|
||||
@@ -779,6 +779,135 @@
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Output tokens/s per GPU, by model",
|
||||
"description": "What each model gets out of each card it holds.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 59,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 22,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) / count by (model) (llmgw_gpu_assigned)",
|
||||
"legendFormat": "{{model}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Output tokens per watt-hour, by model",
|
||||
"description": "Tokens for the energy the model's cards draw.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 59,
|
||||
"w": 12,
|
||||
"h": 7
|
||||
},
|
||||
"id": 23,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) * 3600 / sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)",
|
||||
"legendFormat": "{{model}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU wear",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"id": 24,
|
||||
"description": "What wears cards out: XID errors in the last 15 minutes, memory rows remapped, PCIe retries an hour. Anything above zero here is worth a look on that card's node page.",
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 66,
|
||||
"w": 24,
|
||||
"h": 7
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_xid_errors[15m]) > 0",
|
||||
"legendFormat": "XID {{node}} gpu {{gpu}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum by (node, gpu) (llmgw_gpu_remapped_rows) > 0",
|
||||
"legendFormat": "rows remapped {{node}} gpu {{gpu}}",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_pcie_replays[1h]) > 0",
|
||||
"legendFormat": "PCIe retries/h {{node}} gpu {{gpu}}",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "increase(llmgw_gpu_nvlink_errors[1h]) > 0",
|
||||
"legendFormat": "NVLink {{kind}}/h {{node}} gpu {{gpu}}",
|
||||
"refId": "D"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -224,6 +224,60 @@ groups:
|
||||
card may be computing on corrupted memory. Drain the node
|
||||
(llmgw ctl drain) and have the card looked at.
|
||||
|
||||
- alert: GPUXidError
|
||||
expr: increase(llmgw_gpu_xid_errors[15m]) > 0
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
|
||||
description: >-
|
||||
The driver reported an error on the card (its code is
|
||||
llmgw_gpu_last_xid, and the node page says it in words). 79 is a
|
||||
card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
|
||||
94/95 contained and uncontained ECC errors: those need the card
|
||||
looked at. Others are often an engine's own fault, and pass.
|
||||
|
||||
- alert: GPUOutOfSpareRows
|
||||
expr: llmgw_gpu_remap_failure == 1
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
|
||||
description: >-
|
||||
The card cannot remap another failing row. Drain the node and
|
||||
have the card replaced.
|
||||
|
||||
- alert: GPUNeedsReset
|
||||
expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
|
||||
description: >-
|
||||
Rows remapped or pages retired take effect only after the card is
|
||||
reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
|
||||
it back.
|
||||
|
||||
- alert: GPUPCIeRetriesClimbing
|
||||
expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
|
||||
description: >-
|
||||
The card's link to the host is retrying. A riser, the slot or a
|
||||
cable is the usual cause, and transfers to the card slow down.
|
||||
|
||||
- alert: GPUNVLinkErrorsClimbing
|
||||
expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
|
||||
for: 0m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
|
||||
description: >-
|
||||
Errors on the card's NVLink links: tensor-parallel replicas on it
|
||||
slow down, and a link that keeps failing takes a replica with it.
|
||||
|
||||
- alert: GPUHot
|
||||
expr: llmgw_gpu_temperature_celsius > 87
|
||||
for: 10m
|
||||
|
||||
+1
-1
@@ -1271,7 +1271,7 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
can see each other, seeding between them is the obvious next thing to build.
|
||||
- **A budget without a price never binds.** Set `default_price` or `pricing`
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
displays rather than something that stops anything. The Limits tab says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
|
||||
no build from 0.20 on has Volta kernels, and the node picks the newest that
|
||||
|
||||
+16
-6
@@ -191,7 +191,8 @@ project or person, the teams and projects with their limits and spend, and every
|
||||
member. Only what was spent in that organization's projects is on it; what its
|
||||
people do elsewhere is not. The page is read-only — a manager is not an
|
||||
administrator of the cluster, and limits and memberships stay with those who
|
||||
are.
|
||||
are. An administrator following `/org` lands on the same organization's panel
|
||||
on the people page instead, where all of that can be changed.
|
||||
|
||||
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
|
||||
will no longer move a team or project that already exists under another parent;
|
||||
@@ -259,7 +260,7 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
|
||||
A model's canonical name and any of its aliases name the same thing, so either
|
||||
form works.
|
||||
|
||||
Budgets are given in whole units, as the quotas page shows them, and what a unit
|
||||
Budgets are given in whole units, as the Limits tab shows them, and what a unit
|
||||
means is yours to decide. Self-hosted models have no per-token price, so it can
|
||||
be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
|
||||
a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
|
||||
@@ -409,10 +410,19 @@ page exists.
|
||||
opening onto its members and its keys. From there an admin approves people
|
||||
(into a project or on their own), adds them to and takes them out of
|
||||
organizations, teams and projects, suspends and removes them, revokes keys, and
|
||||
adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
|
||||
sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
|
||||
bottom. The page works as plain forms; with scripts on, a change updates it in
|
||||
place and the projects you had open stay open.
|
||||
adds, renames, moves and removes organizations, teams and projects. A
|
||||
searchable list of every account is at the bottom. Choosing any organization,
|
||||
team, project or person opens a panel beside the tree with its limits, its
|
||||
capacity quota and its use over the last week, and its limits can be changed
|
||||
there.
|
||||
|
||||
The same page has two more tabs, each at the address it had as a page of its
|
||||
own: **Limits** (`/admin/quotas`) is every limit in the tree at once — per
|
||||
branch, per member of any part of it, and per person — with the capacity quotas;
|
||||
**Usage** (`/admin/usage`) is the cluster's use over time by organization, team,
|
||||
project or person, and the GPU time each owner held. The page works as plain
|
||||
forms; with scripts on, a change updates it in place and the projects you had
|
||||
open stay open.
|
||||
|
||||
Three things are deliberately not there:
|
||||
|
||||
|
||||
+43
-4
@@ -104,6 +104,8 @@ type Agent struct {
|
||||
// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
|
||||
// from nvidia-smi.
|
||||
sensors *discovery.SensorSampler
|
||||
// wear reads what wears a GPU out: remapped rows, PCIe retries, XIDs.
|
||||
wear *discovery.HealthSampler
|
||||
|
||||
// fetchWatchers are the shards waiting on each source, so progress on one
|
||||
// download reaches every shard that asked for it. A fetch is single-flight
|
||||
@@ -150,6 +152,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
provKick: make(chan struct{}, 1),
|
||||
load: discovery.NewLoadSampler(),
|
||||
sensors: discovery.NewSensorSampler(),
|
||||
wear: discovery.NewHealthSampler(),
|
||||
}
|
||||
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
|
||||
return a, nil
|
||||
@@ -588,8 +591,24 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
|
||||
}
|
||||
a.mu.Unlock()
|
||||
|
||||
for _, g := range inv.GPUs {
|
||||
hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
|
||||
// Measured now, not carried from registration: a card is shared with
|
||||
// whatever else runs on the machine, and that comes and goes while the
|
||||
// agent runs. The registration figure stays only for a node where
|
||||
// nvidia-smi cannot be asked.
|
||||
if len(inv.GPUs) > 0 {
|
||||
var shards []int
|
||||
for _, st := range a.sup.Statuses() {
|
||||
if st.PID > 0 {
|
||||
shards = append(shards, st.PID)
|
||||
}
|
||||
}
|
||||
hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = a.foreignGPUMemory(context.Background(), shards)
|
||||
}
|
||||
if len(hb.GpuMemoryUsedMb) != len(inv.GPUs) {
|
||||
hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = nil, nil
|
||||
for _, g := range inv.GPUs {
|
||||
hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
|
||||
}
|
||||
}
|
||||
|
||||
// Rescanned here rather than reused from registration. Disk is not a fact
|
||||
@@ -656,14 +675,34 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
|
||||
if a.sensors != nil {
|
||||
se := a.sensors.Sample(context.Background())
|
||||
out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
|
||||
var wear map[int]discovery.GPUWear
|
||||
if a.wear != nil && len(se.GPUs) > 0 {
|
||||
wear = map[int]discovery.GPUWear{}
|
||||
for _, w := range a.wear.Sample(context.Background()) {
|
||||
wear[w.Index] = w
|
||||
}
|
||||
}
|
||||
for _, g := range se.GPUs {
|
||||
out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
|
||||
gs := &llmgwv1.GPUSensor{
|
||||
Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
|
||||
PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
|
||||
UtilizationPercent: g.UtilPercent, MemoryUtilizationPercent: g.MemUtilPercent,
|
||||
MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
|
||||
SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
|
||||
})
|
||||
}
|
||||
if w, ok := wear[g.Index]; ok {
|
||||
gs.HasWear = true
|
||||
gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
|
||||
w.PCIeReplays, w.RetiredSBE, w.RetiredDBE, w.RetirePending
|
||||
gs.RemappedRowsCorrectable, gs.RemappedRowsUncorrectable, gs.RemapPending, gs.RemapFailure =
|
||||
w.RemappedCorrectable, w.RemappedUncorrectable, w.RemapPending, w.RemapFailure
|
||||
gs.NvlinkCrcErrors, gs.NvlinkReplayErrors, gs.NvlinkRecoveryErrors = w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery
|
||||
gs.XidErrors, gs.LastXid, gs.LastXidText = w.XIDs, int32(w.LastXID), w.LastXIDText
|
||||
if !w.LastXIDAt.IsZero() {
|
||||
gs.LastXidUnix = w.LastXIDAt.Unix()
|
||||
}
|
||||
}
|
||||
out.Gpus = append(out.Gpus, gs)
|
||||
}
|
||||
}
|
||||
return out
|
||||
|
||||
@@ -106,9 +106,27 @@ func parseSMI(out string) ([]GPU, error) {
|
||||
return gpus, nil
|
||||
}
|
||||
|
||||
// residentMemoryToleranceMB is how much allocated memory is written off as
|
||||
// ForeignMemoryToleranceMB is how much allocated memory is written off as
|
||||
// driver overhead rather than a foreign process.
|
||||
const residentMemoryToleranceMB = 512
|
||||
const ForeignMemoryToleranceMB = 512
|
||||
|
||||
// foreignMemorySuffix ends the problem a card with another tenant on it is
|
||||
// kept out of placement for. One phrase, so that the control plane can find
|
||||
// it again and drop it when the tenant leaves.
|
||||
const foreignMemorySuffix = " MiB already allocated by another process"
|
||||
|
||||
// ForeignMemoryProblem is the problem for mb of memory held by something
|
||||
// that is not one of this agent's engines, or "" when that is too little to
|
||||
// count.
|
||||
func ForeignMemoryProblem(mb int) string {
|
||||
if mb <= ForeignMemoryToleranceMB {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf("%d%s", mb, foreignMemorySuffix)
|
||||
}
|
||||
|
||||
// IsForeignMemoryProblem reports whether a problem is ForeignMemoryProblem's.
|
||||
func IsForeignMemoryProblem(p string) bool { return strings.HasSuffix(p, foreignMemorySuffix) }
|
||||
|
||||
// assessHealth flags conditions that should keep a GPU out of placement.
|
||||
func assessHealth(g *GPU) {
|
||||
@@ -119,8 +137,12 @@ func assessHealth(g *GPU) {
|
||||
// Memory already in use means something else is on the card. Starting an
|
||||
// engine there would either fail to allocate or quietly evict the other
|
||||
// process's throughput.
|
||||
if g.MemoryUsedMB > residentMemoryToleranceMB {
|
||||
g.Health.Problems = append(g.Health.Problems,
|
||||
fmt.Sprintf("%d MiB already allocated by another process", g.MemoryUsedMB))
|
||||
//
|
||||
// Only a first reading: this runs before the agent knows which processes
|
||||
// are its own, so an engine still running from before a restart reads as
|
||||
// foreign here. Every heartbeat measures it again, without the agent's
|
||||
// own engines, and the control plane replaces this with that.
|
||||
if p := ForeignMemoryProblem(g.MemoryUsedMB); p != "" {
|
||||
g.Health.Problems = append(g.Health.Problems, p)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// GPUWear is what wears a card out or takes it down, beyond temperature:
|
||||
// read every minute, from nvidia-smi and the kernel's log. The early signs of
|
||||
// a card failing are here — a growing count of rows its memory had to remap,
|
||||
// PCIe link retries, XID errors — long before a replica on it fails.
|
||||
//
|
||||
// A count of -1 is not reported: a card without row remapping (before
|
||||
// Ampere) has no such count, which is not the same as none.
|
||||
type GPUWear struct {
|
||||
Index int
|
||||
// PCIeReplays are link-level retries on the card's PCIe link: a few are
|
||||
// noise, a count that keeps climbing is a riser, slot or cable.
|
||||
PCIeReplays int64
|
||||
// RetiredSBE and RetiredDBE are memory pages retired for single and
|
||||
// double bit errors (cards before Ampere); RetirePending is a page
|
||||
// waiting for a reset to be retired.
|
||||
RetiredSBE, RetiredDBE int64
|
||||
RetirePending bool
|
||||
// RemappedCorrectable and RemappedUncorrectable are memory rows remapped
|
||||
// (Ampere and later). RemapPending waits for a reset; RemapFailure is a
|
||||
// card out of spare rows — it needs replacing.
|
||||
RemappedCorrectable, RemappedUncorrectable int64
|
||||
RemapPending, RemapFailure bool
|
||||
// NVLink errors summed over the card's links, by kind.
|
||||
NVLinkCRC, NVLinkReplay, NVLinkRecovery int64
|
||||
// XIDs are the driver's error events for this card since the agent
|
||||
// started, and the last of them.
|
||||
XIDs int64
|
||||
LastXID int
|
||||
LastXIDText string
|
||||
LastXIDAt time.Time
|
||||
}
|
||||
|
||||
// healthEvery is how often health is read: it changes over hours, and the
|
||||
// kernel log and nvidia-smi's full report are the heavier reads.
|
||||
const healthEvery = time.Minute
|
||||
|
||||
// HealthSampler reads GPUWear, keeping the XID counts between reads.
|
||||
type HealthSampler struct {
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
at time.Time
|
||||
last []GPUWear
|
||||
since time.Time
|
||||
xids map[int]*GPUWear
|
||||
started bool
|
||||
}
|
||||
|
||||
// NewHealthSampler reads the machine's own nvidia-smi and kernel log.
|
||||
func NewHealthSampler() *HealthSampler {
|
||||
return &HealthSampler{now: time.Now, xids: map[int]*GPUWear{},
|
||||
run: func(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
return exec.CommandContext(ctx, name, args...).Output() //nolint:gosec // fixed tools
|
||||
}}
|
||||
}
|
||||
|
||||
// Sample is the cards' health, read again once healthEvery has passed. Nil
|
||||
// on a machine without nvidia-smi.
|
||||
func (h *HealthSampler) Sample(ctx context.Context) []GPUWear {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
now := h.now()
|
||||
if !h.at.IsZero() && now.Sub(h.at) < healthEvery {
|
||||
return h.last
|
||||
}
|
||||
h.at = now
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
report, err := h.run(ctx, "nvidia-smi", "-q", "-x")
|
||||
if err != nil {
|
||||
h.last = nil
|
||||
return nil
|
||||
}
|
||||
cards, bus := parseHealthReport(report)
|
||||
if out, err := h.run(ctx, "nvidia-smi", "nvlink", "-e"); err == nil {
|
||||
for i, e := range parseNVLinkErrors(string(out)) {
|
||||
if i < len(cards) {
|
||||
cards[i].NVLinkCRC, cards[i].NVLinkReplay, cards[i].NVLinkRecovery = e[0], e[1], e[2]
|
||||
}
|
||||
}
|
||||
}
|
||||
h.readXIDs(ctx, now, bus)
|
||||
for i := range cards {
|
||||
if x := h.xids[i]; x != nil {
|
||||
cards[i].XIDs, cards[i].LastXID, cards[i].LastXIDText, cards[i].LastXIDAt = x.XIDs, x.LastXID, x.LastXIDText, x.LastXIDAt
|
||||
}
|
||||
}
|
||||
h.last = cards
|
||||
return cards
|
||||
}
|
||||
|
||||
// readXIDs adds the XID events logged since the last read. The first read
|
||||
// starts from now: what happened before the agent started was before, and
|
||||
// counting the whole log would raise every alert on every restart.
|
||||
func (h *HealthSampler) readXIDs(ctx context.Context, now time.Time, bus map[string]int) {
|
||||
if !h.started {
|
||||
h.started, h.since = true, now
|
||||
return
|
||||
}
|
||||
since := h.since
|
||||
h.since = now
|
||||
out, err := h.run(ctx, "journalctl", "-k", "--no-pager", "-o", "short-unix",
|
||||
"--since", "@"+strconv.FormatInt(since.Unix(), 10), "-g", "NVRM: Xid")
|
||||
if err != nil {
|
||||
// No journal, or nothing matched (journalctl exits 1 for that).
|
||||
return
|
||||
}
|
||||
for _, e := range parseXIDs(string(out)) {
|
||||
i, ok := bus[e.bus]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
x := h.xids[i]
|
||||
if x == nil {
|
||||
x = &GPUWear{Index: i}
|
||||
h.xids[i] = x
|
||||
}
|
||||
x.XIDs++
|
||||
x.LastXID, x.LastXIDText = e.code, e.text
|
||||
x.LastXIDAt = now
|
||||
if e.at > 0 {
|
||||
x.LastXIDAt = time.Unix(int64(e.at), 0)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// healthLog is the part of nvidia-smi's XML report health is read from.
|
||||
type healthLog struct {
|
||||
GPUs []struct {
|
||||
ID string `xml:"id,attr"`
|
||||
PCI struct {
|
||||
Replay string `xml:"replay_counter"`
|
||||
} `xml:"pci"`
|
||||
Retired struct {
|
||||
SBE string `xml:"multiple_single_bit_retirement>retired_count"`
|
||||
DBE string `xml:"double_bit_retirement>retired_count"`
|
||||
Pending string `xml:"pending_retirement"`
|
||||
// Older drivers name it pending_blacklist.
|
||||
Blacklist string `xml:"pending_blacklist"`
|
||||
} `xml:"retired_pages"`
|
||||
Remapped struct {
|
||||
Corr string `xml:"remapped_row_corr"`
|
||||
Unc string `xml:"remapped_row_unc"`
|
||||
Pending string `xml:"remapped_row_pending"`
|
||||
Failure string `xml:"remapped_row_failure"`
|
||||
} `xml:"remapped_rows"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseHealthReport reads each card's health, in index order, and maps each
|
||||
// card's PCI bus ("3b:00", as the kernel names it) to its index.
|
||||
func parseHealthReport(report []byte) ([]GPUWear, map[string]int) {
|
||||
var log healthLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
bus := map[string]int{}
|
||||
out := make([]GPUWear, 0, len(log.GPUs))
|
||||
for i, g := range log.GPUs {
|
||||
bus[busOf(g.ID)] = i
|
||||
out = append(out, GPUWear{
|
||||
Index: i,
|
||||
PCIeReplays: count(g.PCI.Replay),
|
||||
RetiredSBE: count(g.Retired.SBE),
|
||||
RetiredDBE: count(g.Retired.DBE),
|
||||
RetirePending: yes(g.Retired.Pending) || yes(g.Retired.Blacklist),
|
||||
RemappedCorrectable: count(g.Remapped.Corr),
|
||||
RemappedUncorrectable: count(g.Remapped.Unc),
|
||||
RemapPending: yes(g.Remapped.Pending),
|
||||
RemapFailure: yes(g.Remapped.Failure),
|
||||
})
|
||||
}
|
||||
return out, bus
|
||||
}
|
||||
|
||||
// busOf is a PCI address as the kernel's XID lines carry it: nvidia-smi
|
||||
// says "00000000:3B:00.0", the kernel "0000:3b:00".
|
||||
func busOf(id string) string {
|
||||
id = strings.ToLower(strings.TrimSpace(id))
|
||||
if i := strings.LastIndex(id, "."); i >= 0 {
|
||||
id = id[:i]
|
||||
}
|
||||
parts := strings.Split(id, ":")
|
||||
if len(parts) >= 2 {
|
||||
return parts[len(parts)-2] + ":" + parts[len(parts)-1]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func count(s string) int64 {
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 64)
|
||||
if err != nil {
|
||||
return -1
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func yes(s string) bool { return strings.EqualFold(strings.TrimSpace(s), "yes") }
|
||||
|
||||
// nvlinkError matches a line of `nvidia-smi nvlink -e`, whose wording varies
|
||||
// by driver: "Link 0: Replay Errors: 0", "Link 0: CRC Errors: 0",
|
||||
// "Link 3: Data CRC Errors: 2".
|
||||
var (
|
||||
nvlinkGPU = regexp.MustCompile(`^GPU (\d+):`)
|
||||
nvlinkError = regexp.MustCompile(`Link \d+:\s*(.*?)\s*Errors?\s*:\s*(\d+)`)
|
||||
)
|
||||
|
||||
// parseNVLinkErrors sums each card's link errors by kind: CRC, replay,
|
||||
// recovery.
|
||||
func parseNVLinkErrors(out string) map[int][3]int64 {
|
||||
res := map[int][3]int64{}
|
||||
gpu := -1
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if m := nvlinkGPU.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
|
||||
gpu, _ = strconv.Atoi(m[1])
|
||||
continue
|
||||
}
|
||||
m := nvlinkError.FindStringSubmatch(line)
|
||||
if m == nil || gpu < 0 {
|
||||
continue
|
||||
}
|
||||
n, _ := strconv.ParseInt(m[2], 10, 64)
|
||||
e := res[gpu]
|
||||
kind := strings.ToLower(m[1])
|
||||
switch {
|
||||
case strings.Contains(kind, "crc"):
|
||||
e[0] += n
|
||||
case strings.Contains(kind, "replay"):
|
||||
e[1] += n
|
||||
case strings.Contains(kind, "recovery"):
|
||||
e[2] += n
|
||||
}
|
||||
res[gpu] = e
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
type xidEvent struct {
|
||||
bus string
|
||||
code int
|
||||
text string
|
||||
at float64
|
||||
}
|
||||
|
||||
// xidLine matches the driver's report of an XID:
|
||||
// "NVRM: Xid (PCI:0000:3b:00): 79, pid=..., GPU has fallen off the bus."
|
||||
var xidLine = regexp.MustCompile(`NVRM: Xid \(PCI:([0-9a-fA-F:.]+)\): (\d+),\s*(.*)$`)
|
||||
|
||||
// parseXIDs reads journalctl's short-unix lines: a timestamp, then the
|
||||
// message.
|
||||
func parseXIDs(out string) []xidEvent {
|
||||
var evs []xidEvent
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
m := xidLine.FindStringSubmatch(line)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
code, _ := strconv.Atoi(m[2])
|
||||
e := xidEvent{bus: busOf(m[1]), code: code, text: strings.TrimSpace(m[3])}
|
||||
if f := strings.Fields(line); len(f) > 0 {
|
||||
e.at, _ = strconv.ParseFloat(f[0], 64)
|
||||
}
|
||||
evs = append(evs, e)
|
||||
}
|
||||
return evs
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const healthXML = `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<gpu id="00000000:3B:00.0">
|
||||
<pci><replay_counter>0</replay_counter></pci>
|
||||
<retired_pages>
|
||||
<multiple_single_bit_retirement><retired_count>N/A</retired_count></multiple_single_bit_retirement>
|
||||
<double_bit_retirement><retired_count>N/A</retired_count></double_bit_retirement>
|
||||
<pending_retirement>N/A</pending_retirement>
|
||||
</retired_pages>
|
||||
<remapped_rows>
|
||||
<remapped_row_corr>2</remapped_row_corr>
|
||||
<remapped_row_unc>1</remapped_row_unc>
|
||||
<remapped_row_pending>Yes</remapped_row_pending>
|
||||
<remapped_row_failure>No</remapped_row_failure>
|
||||
</remapped_rows>
|
||||
</gpu>
|
||||
<gpu id="00000000:5E:00.0">
|
||||
<pci><replay_counter>4021</replay_counter></pci>
|
||||
<retired_pages>
|
||||
<multiple_single_bit_retirement><retired_count>3</retired_count></multiple_single_bit_retirement>
|
||||
<double_bit_retirement><retired_count>1</retired_count></double_bit_retirement>
|
||||
<pending_blacklist>Yes</pending_blacklist>
|
||||
</retired_pages>
|
||||
<remapped_rows>N/A</remapped_rows>
|
||||
</gpu>
|
||||
</nvidia_smi_log>`
|
||||
|
||||
const nvlinkOut = `GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-aaa)
|
||||
Link 0: Replay Errors: 1
|
||||
Link 0: Recovery Errors: 0
|
||||
Link 0: CRC Errors: 5
|
||||
Link 1: Data CRC Errors: 2
|
||||
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-bbb)
|
||||
Link 0: Replay Errors: 0
|
||||
`
|
||||
|
||||
func TestAGPUsWearAndFailuresAreReadAndCountedFromWhenTheAgentStarted(t *testing.T) {
|
||||
now := time.Unix(1_800_000_000, 0)
|
||||
journal := ""
|
||||
h := &HealthSampler{now: func() time.Time { return now }, xids: map[int]*GPUWear{},
|
||||
run: func(_ context.Context, name string, args ...string) ([]byte, error) {
|
||||
switch {
|
||||
case name == "nvidia-smi" && args[0] == "-q":
|
||||
return []byte(healthXML), nil
|
||||
case name == "nvidia-smi" && args[0] == "nvlink":
|
||||
return []byte(nvlinkOut), nil
|
||||
case name == "journalctl":
|
||||
if journal == "" {
|
||||
return nil, errors.New("exit status 1")
|
||||
}
|
||||
return []byte(journal), nil
|
||||
}
|
||||
return nil, errors.New("unexpected " + name)
|
||||
}}
|
||||
got := h.Sample(t.Context())
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
a, b := got[0], got[1]
|
||||
if a.RemappedCorrectable != 2 || a.RemappedUncorrectable != 1 || !a.RemapPending || a.RemapFailure ||
|
||||
a.RetiredSBE != -1 || a.PCIeReplays != 0 || a.NVLinkCRC != 7 || a.NVLinkReplay != 1 {
|
||||
t.Errorf("card 0 %+v", a)
|
||||
}
|
||||
if b.PCIeReplays != 4021 || b.RetiredSBE != 3 || b.RetiredDBE != 1 || !b.RetirePending || b.RemappedCorrectable != -1 {
|
||||
t.Errorf("card 1 %+v", b)
|
||||
}
|
||||
|
||||
// An XID on card 1, logged after the agent started; read on the next
|
||||
// sample a minute on.
|
||||
journal = "1800000030.123456 gpu07 kernel: NVRM: Xid (PCI:0000:5e:00): 79, pid=1234, name=llama-server, GPU has fallen off the bus.\n"
|
||||
now = now.Add(61 * time.Second)
|
||||
got = h.Sample(t.Context())
|
||||
if got[1].XIDs != 1 || got[1].LastXID != 79 || !strings.Contains(got[1].LastXIDText, "fallen off the bus") ||
|
||||
got[1].LastXIDAt.Unix() != 1_800_000_030 || got[0].XIDs != 0 {
|
||||
t.Errorf("after an XID: %+v / %+v", got[0], got[1])
|
||||
}
|
||||
// Within the minute, the last reading.
|
||||
journal = ""
|
||||
now = now.Add(10 * time.Second)
|
||||
if again := h.Sample(t.Context()); again[1].XIDs != 1 {
|
||||
t.Errorf("%+v", again[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestBusIDsAreReadAsTheKernelWritesThem(t *testing.T) {
|
||||
for in, want := range map[string]string{"00000000:3B:00.0": "3b:00", "0000:3b:00": "3b:00", "0000:5e:00.0": "5e:00"} {
|
||||
if got := busOf(in); got != want {
|
||||
t.Errorf("%s: %s, want %s", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -34,13 +34,47 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
return parseGPUProcesses(out)
|
||||
}
|
||||
|
||||
// GPUMemory is one card's memory as nvidia-smi reports it at a moment: what
|
||||
// is allocated on it, and which processes hold how much of that.
|
||||
type GPUMemory struct {
|
||||
UsedMB int
|
||||
Procs []GPUProcess
|
||||
}
|
||||
|
||||
// GPUProcess is one process holding memory on a card.
|
||||
type GPUProcess struct {
|
||||
PID int
|
||||
UsedMB int
|
||||
}
|
||||
|
||||
// GPUMemoryByProcess reports each GPU's memory, by index, with the processes
|
||||
// holding it — from the same report as GPUProcesses, so the two never
|
||||
// disagree about who is on a card.
|
||||
//
|
||||
// A node with no nvidia-smi returns nil and no error, as discovery does.
|
||||
func GPUMemoryByProcess(ctx context.Context, binary string) ([]GPUMemory, error) {
|
||||
if binary == "" {
|
||||
if _, err := exec.LookPath("nvidia-smi"); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
binary = "nvidia-smi"
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseGPUMemory(out)
|
||||
}
|
||||
|
||||
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
|
||||
// index order, and the processes on it.
|
||||
// index order, its memory and the processes on it.
|
||||
type smiLog struct {
|
||||
GPUs []struct {
|
||||
Used string `xml:"fb_memory_usage>used"`
|
||||
Processes []struct {
|
||||
PID string `xml:"pid"`
|
||||
Type string `xml:"type"`
|
||||
PID string `xml:"pid"`
|
||||
Type string `xml:"type"`
|
||||
UsedMB string `xml:"used_memory"`
|
||||
} `xml:"processes>process_info"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
@@ -48,17 +82,39 @@ type smiLog struct {
|
||||
// parseGPUProcesses reads the report. GPUs are listed in index order, which
|
||||
// is nvidia-smi's own numbering — the one placement assigns by.
|
||||
func parseGPUProcesses(report []byte) (map[int][]int, error) {
|
||||
mem, err := parseGPUMemory(report)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[int][]int{}
|
||||
for i, g := range mem {
|
||||
for _, p := range g.Procs {
|
||||
out[i] = append(out[i], p.PID)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// parseGPUMemory reads the report into each GPU's memory and its holders.
|
||||
func parseGPUMemory(report []byte) ([]GPUMemory, error) {
|
||||
var log smiLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
out := map[int][]int{}
|
||||
out := make([]GPUMemory, len(log.GPUs))
|
||||
for i, g := range log.GPUs {
|
||||
out[i].UsedMB = mib(g.Used)
|
||||
for _, p := range g.Processes {
|
||||
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
|
||||
out[i] = append(out[i], pid)
|
||||
out[i].Procs = append(out[i].Procs, GPUProcess{PID: pid, UsedMB: mib(p.UsedMB)})
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// mib reads "4932 MiB"; anything else, "N/A" included, is zero.
|
||||
func mib(s string) int {
|
||||
n, _ := strconv.Atoi(strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(s), "MiB")))
|
||||
return n
|
||||
}
|
||||
@@ -36,3 +36,43 @@ func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
|
||||
t.Error("a report that does not parse was read as no processes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGPUMemoryIsReadPerProcess(t *testing.T) {
|
||||
report := `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<gpu id="00000000:01:00.0"><product_name>Quadro RTX 6000</product_name>
|
||||
<fb_memory_usage><total>24576 MiB</total><used>14514 MiB</used><free>10062 MiB</free></fb_memory_usage>
|
||||
<processes>
|
||||
<process_info><pid>71315</pid><type>C</type><used_memory>4932 MiB</used_memory></process_info>
|
||||
<process_info><pid>4100</pid><type>C</type><used_memory>9500 MiB</used_memory></process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:02:00.0">
|
||||
<fb_memory_usage><used>N/A</used></fb_memory_usage>
|
||||
<processes></processes>
|
||||
</gpu>
|
||||
</nvidia_smi_log>`
|
||||
got, err := parseGPUMemory([]byte(report))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []GPUMemory{
|
||||
{UsedMB: 14514, Procs: []GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 4100, UsedMB: 9500}}},
|
||||
{},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestForeignMemoryBelowTheToleranceIsNoProblem(t *testing.T) {
|
||||
// A CUDA context that has let go of its model still holds a few hundred
|
||||
// MiB: ComfyUI after /free held 324.
|
||||
if p := ForeignMemoryProblem(330); p != "" {
|
||||
t.Errorf("330 MiB is a problem: %q", p)
|
||||
}
|
||||
p := ForeignMemoryProblem(5014)
|
||||
if !IsForeignMemoryProblem(p) || IsForeignMemoryProblem("2 uncorrected ECC errors") {
|
||||
t.Errorf("%q is not recognised as the foreign memory problem, or ECC is", p)
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
// nvidia-smi and /proc.
|
||||
var (
|
||||
gpuProcesses = discovery.GPUProcesses
|
||||
gpuMemory = discovery.GPUMemoryByProcess
|
||||
processGroup = pgidOf
|
||||
// gpuCheckWait is how long a shard has, once it answers, to show up on
|
||||
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
|
||||
@@ -75,6 +76,44 @@ func (a *Agent) onItsGPUs(ctx context.Context, eng engines.Engine, spec engines.
|
||||
"or run `llmgw doctor` on this node", joinIndices(missing), seen)
|
||||
}
|
||||
|
||||
// foreignGPUMemory is what each GPU holds for anything other than this
|
||||
// agent's shards, by index, and each GPU's memory in use. Nil when nvidia-smi
|
||||
// cannot be asked, which is no reading rather than an empty card.
|
||||
//
|
||||
// Everything allocated less what the shards' processes hold, rather than the
|
||||
// sum of the other processes: a process in another container's PID namespace
|
||||
// holds memory nvidia-smi lists under no PID this node can see, and it is
|
||||
// still not memory placement can hand out.
|
||||
func (a *Agent) foreignGPUMemory(ctx context.Context, shards []int) (foreign, used []int32) {
|
||||
mem, err := gpuMemory(ctx, a.cfg.Discovery.NvidiaSMI)
|
||||
if err != nil || mem == nil {
|
||||
return nil, nil
|
||||
}
|
||||
groups := make([]int, len(shards))
|
||||
for i, pid := range shards {
|
||||
groups[i] = processGroup(pid)
|
||||
}
|
||||
ours := func(p int) bool {
|
||||
for i, pid := range shards {
|
||||
if holds([]int{p}, pid, groups[i]) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
for _, g := range mem {
|
||||
f := g.UsedMB
|
||||
for _, p := range g.Procs {
|
||||
if ours(p.PID) {
|
||||
f -= p.UsedMB
|
||||
}
|
||||
}
|
||||
foreign = append(foreign, int32(max(f, 0)))
|
||||
used = append(used, int32(g.UsedMB))
|
||||
}
|
||||
return foreign, used
|
||||
}
|
||||
|
||||
// holds reports whether any of pids is the shard's process or in its group:
|
||||
// an engine that computes in a child process is still the shard.
|
||||
func holds(pids []int, pid, group int) bool {
|
||||
|
||||
@@ -3,10 +3,12 @@ package agent
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
@@ -89,3 +91,34 @@ func TestTheCheckWaitsForTheShardToShowUpOnItsGPU(t *testing.T) {
|
||||
t.Fatalf("nvidia-smi lagging by a second failed the shard: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestForeignGPUMemoryLeavesOutTheAgentsOwnShards(t *testing.T) {
|
||||
// cuda01: ComfyUI and one of our engines on the same card. Only ComfyUI's
|
||||
// share — and whatever no listed process accounts for — is foreign.
|
||||
oldMem, oldGroup := gpuMemory, processGroup
|
||||
gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) {
|
||||
return []discovery.GPUMemory{
|
||||
{UsedMB: 15000, Procs: []discovery.GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 512, UsedMB: 9500}}},
|
||||
{UsedMB: 300},
|
||||
}, nil
|
||||
}
|
||||
processGroup = func(pid int) int {
|
||||
if pid >= 500 && pid < 600 {
|
||||
return 500
|
||||
}
|
||||
return pid
|
||||
}
|
||||
t.Cleanup(func() { gpuMemory, processGroup = oldMem, oldGroup })
|
||||
|
||||
a := &Agent{log: slog.Default()}
|
||||
foreign, used := a.foreignGPUMemory(context.Background(), []int{500})
|
||||
if !slices.Equal(foreign, []int32{5500, 300}) || !slices.Equal(used, []int32{15000, 300}) {
|
||||
t.Errorf("foreign %v, used %v; want [5500 300] and [15000 300]", foreign, used)
|
||||
}
|
||||
|
||||
// No nvidia-smi to ask is no reading, not an empty card.
|
||||
gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) { return nil, nil }
|
||||
if foreign, _ := a.foreignGPUMemory(context.Background(), nil); foreign != nil {
|
||||
t.Errorf("no reading came back as %v", foreign)
|
||||
}
|
||||
}
|
||||
@@ -283,3 +283,36 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
|
||||
t.Errorf("the agent's refusal was not passed on: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
|
||||
// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
|
||||
// was kept out of placement. ComfyUI let go of it and the card stayed
|
||||
// out, saying "5014 MiB already allocated by another process", until the
|
||||
// agent was restarted.
|
||||
s := newTestService()
|
||||
nodeWith(s, inventory.Node{
|
||||
ID: "n1", State: inventory.Ready,
|
||||
GPUs: []discovery.GPU{{Index: 0, Health: discovery.GPUHealth{Problems: []string{
|
||||
discovery.ForeignMemoryProblem(5014), "2 uncorrected ECC errors"}}}},
|
||||
})
|
||||
|
||||
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{330}})
|
||||
n, _ := s.inv.Get("n1")
|
||||
if got := n.GPUs[0].Health.Problems; len(got) != 1 || got[0] != "2 uncorrected ECC errors" {
|
||||
t.Fatalf("problems are %q: the tenant left, and only the ECC errors should remain", got)
|
||||
}
|
||||
|
||||
// And the other way: a tenant arriving after registration.
|
||||
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{12000}})
|
||||
n, _ = s.inv.Get("n1")
|
||||
if got := n.GPUs[0].Health.Problems; len(got) != 2 || got[1] != discovery.ForeignMemoryProblem(12000) {
|
||||
t.Fatalf("problems are %q, want the new tenant's 12000 MiB", got)
|
||||
}
|
||||
|
||||
// An agent that sends no reading changes nothing.
|
||||
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuMemoryUsedMb: []int32{100}})
|
||||
n, _ = s.inv.Get("n1")
|
||||
if len(n.GPUs[0].Health.Problems) != 2 {
|
||||
t.Errorf("a heartbeat with no foreign reading cleared the card: %q", n.GPUs[0].Health.Problems)
|
||||
}
|
||||
}
|
||||
@@ -377,11 +377,12 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
|
||||
// a reconnect.
|
||||
func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
used := hb.GetGpuMemoryUsedMb()
|
||||
foreign := hb.GetGpuForeignMemoryMb()
|
||||
problems := hb.GetNewProblems()
|
||||
storage := hb.GetStorage()
|
||||
memMB := int(hb.GetMemoryTotalMb())
|
||||
cpus := int(hb.GetCpuCount())
|
||||
if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
|
||||
if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 &&
|
||||
!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
|
||||
!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
|
||||
!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
|
||||
@@ -397,6 +398,20 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
n.GPUs[i].MemoryUsedMB = int(used[i])
|
||||
}
|
||||
}
|
||||
// Another tenant on a card, judged afresh: the reading at registration is
|
||||
// replaced, both ways. Without this a card that had something on it when
|
||||
// its agent started stayed out of placement after it left, and said so,
|
||||
// until the agent was restarted. Only a reading for every card replaces
|
||||
// it; an agent that sends none, or could not measure, leaves it standing.
|
||||
if len(foreign) == len(n.GPUs) {
|
||||
for i := range n.GPUs {
|
||||
ps := slices.DeleteFunc(slices.Clone(n.GPUs[i].Health.Problems), discovery.IsForeignMemoryProblem)
|
||||
if p := discovery.ForeignMemoryProblem(int(foreign[i])); p != "" {
|
||||
ps = append(ps, p)
|
||||
}
|
||||
n.GPUs[i].Health.Problems = ps
|
||||
}
|
||||
}
|
||||
if len(problems) > 0 {
|
||||
// Problems are reported per node, not per GPU, so attach them to the
|
||||
// node's first GPU rather than guessing which one is affected. Any
|
||||
@@ -588,6 +603,7 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
|
||||
UtilPercent: g.GetUtilizationPercent(), MemUtilPercent: g.GetMemoryUtilizationPercent(),
|
||||
MemoryUsedMB: g.GetMemoryUsedMb(), MemoryTotalMB: g.GetMemoryTotalMb(),
|
||||
SMClockMHz: g.GetSmClockMhz(), ECCUncorrected: g.GetEccUncorrected(),
|
||||
Wear: wearFrom(g),
|
||||
})
|
||||
}
|
||||
return out
|
||||
@@ -964,3 +980,22 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
|
||||
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
|
||||
}
|
||||
}
|
||||
|
||||
// wearFrom is a card's wear, when the agent read it.
|
||||
func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
|
||||
if !g.GetHasWear() {
|
||||
return nil
|
||||
}
|
||||
w := &inventory.GPUWear{
|
||||
PCIeReplays: g.GetPcieReplays(), RetiredSBE: g.GetRetiredPagesSbe(), RetiredDBE: g.GetRetiredPagesDbe(),
|
||||
RetirePending: g.GetRetirementPending(), RemappedCorrectable: g.GetRemappedRowsCorrectable(),
|
||||
RemappedUncorrectable: g.GetRemappedRowsUncorrectable(), RemapPending: g.GetRemapPending(),
|
||||
RemapFailure: g.GetRemapFailure(), NVLinkCRC: g.GetNvlinkCrcErrors(), NVLinkReplay: g.GetNvlinkReplayErrors(),
|
||||
NVLinkRecovery: g.GetNvlinkRecoveryErrors(), XIDs: g.GetXidErrors(), LastXID: int(g.GetLastXid()),
|
||||
LastXIDText: g.GetLastXidText(),
|
||||
}
|
||||
if u := g.GetLastXidUnix(); u > 0 {
|
||||
w.LastXIDAt = time.Unix(u, 0)
|
||||
}
|
||||
return w
|
||||
}
|
||||
@@ -417,3 +417,17 @@ func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
|
||||
|
||||
// EngineChoices are the engine builds chosen per node, or nil.
|
||||
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
|
||||
|
||||
// ModelOwners maps each model to the part of the tenancy tree its capacity
|
||||
// counts against; a model owned by nobody is not in it.
|
||||
func (c *Controller) ModelOwners() map[string]string {
|
||||
out := map[string]string{}
|
||||
if cat := c.Catalog(); cat != nil {
|
||||
for _, m := range cat.Models() {
|
||||
if m.Owner != "" {
|
||||
out[m.Name] = m.Owner
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -73,6 +73,16 @@ type Collector struct {
|
||||
gpuMemoryTotal *prometheus.Desc
|
||||
gpuMemoryUsed *prometheus.Desc
|
||||
gpuUtil *prometheus.Desc
|
||||
gpuAssigned *prometheus.Desc
|
||||
gpuPCIeReplay *prometheus.Desc
|
||||
gpuRetired *prometheus.Desc
|
||||
gpuRetirePend *prometheus.Desc
|
||||
gpuRemapped *prometheus.Desc
|
||||
gpuRemapPend *prometheus.Desc
|
||||
gpuRemapFail *prometheus.Desc
|
||||
gpuNVLinkErr *prometheus.Desc
|
||||
gpuXID *prometheus.Desc
|
||||
gpuLastXID *prometheus.Desc
|
||||
gpuMemUtil *prometheus.Desc
|
||||
gpuSMClock *prometheus.Desc
|
||||
gpuTemp *prometheus.Desc
|
||||
@@ -151,6 +161,28 @@ func New(src Source) *Collector {
|
||||
"How busy a GPU was over the last sample period. Absent when the driver does not "+
|
||||
"report it, which is not the same as idle. Memory used says a model is loaded; "+
|
||||
"this says whether it is doing anything.", gpu, nil),
|
||||
gpuPCIeReplay: prometheus.NewDesc("llmgw_gpu_pcie_replays",
|
||||
"PCIe link retries on a GPU's link since the driver loaded; one that keeps climbing is a riser, slot or cable.", gpu, nil),
|
||||
gpuRetired: prometheus.NewDesc("llmgw_gpu_retired_pages",
|
||||
"Memory pages retired, by cause (sbe, dbe), on GPUs before Ampere.", append(gpu, "cause"), nil),
|
||||
gpuRetirePend: prometheus.NewDesc("llmgw_gpu_retirement_pending",
|
||||
"1 when retired pages wait for a GPU reset to take effect.", gpu, nil),
|
||||
gpuRemapped: prometheus.NewDesc("llmgw_gpu_remapped_rows",
|
||||
"Memory rows remapped, by kind (correctable, uncorrectable), on Ampere and later.", append(gpu, "kind"), nil),
|
||||
gpuRemapPend: prometheus.NewDesc("llmgw_gpu_remap_pending",
|
||||
"1 when remapped rows wait for a GPU reset to take effect.", gpu, nil),
|
||||
gpuRemapFail: prometheus.NewDesc("llmgw_gpu_remap_failure",
|
||||
"1 when a GPU has run out of spare memory rows: it needs replacing.", gpu, nil),
|
||||
gpuNVLinkErr: prometheus.NewDesc("llmgw_gpu_nvlink_errors",
|
||||
"NVLink errors summed over a GPU's links, by kind (crc, replay, recovery).", append(gpu, "kind"), nil),
|
||||
gpuXID: prometheus.NewDesc("llmgw_gpu_xid_errors",
|
||||
"The driver's XID error events for a GPU since its node's agent started.", gpu, nil),
|
||||
gpuLastXID: prometheus.NewDesc("llmgw_gpu_last_xid",
|
||||
"The code of a GPU's last XID error since its agent started; absent when there was none.", gpu, nil),
|
||||
gpuAssigned: prometheus.NewDesc("llmgw_gpu_assigned",
|
||||
"1 for each GPU a replica holds: which model, which replica, and whose quota it counts against. "+
|
||||
"Joined on node and gpu with the other llmgw_gpu_ series, it says what each model's cards are doing.",
|
||||
[]string{"node", "gpu", "model", "replica", "owner"}, nil),
|
||||
gpuMemUtil: prometheus.NewDesc("llmgw_gpu_memory_utilization_percent",
|
||||
"How much of the sample period a GPU's memory was being read or written: "+
|
||||
"decode is bound by it, so a card busy here and idle above is waiting on memory.", gpu, nil),
|
||||
@@ -278,7 +310,9 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
|
||||
c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
|
||||
c.gpuNVLJoined, c.imexUp,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuPCIeReplay, c.gpuRetired, c.gpuRetirePend, c.gpuRemapped, c.gpuRemapPend, c.gpuRemapFail,
|
||||
c.gpuNVLinkErr, c.gpuXID, c.gpuLastXID,
|
||||
c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
|
||||
c.gpuNVLink, c.gpuECC,
|
||||
c.replicaState, c.replicaReady, c.replicaIdle,
|
||||
@@ -294,6 +328,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
now := time.Now()
|
||||
c.collectQuotas(ch)
|
||||
c.collectAssignments(ch)
|
||||
|
||||
loads := map[string]*inventory.Load{}
|
||||
for _, n := range c.src.Nodes() {
|
||||
@@ -539,6 +574,12 @@ func b2f(b bool) float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
// OwnerSource says whose quota each model counts against, for labelling
|
||||
// what its replicas hold. Asked for by type assertion.
|
||||
type OwnerSource interface {
|
||||
ModelOwners() map[string]string
|
||||
}
|
||||
|
||||
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
|
||||
// assertion, like HealthSource.
|
||||
type QuotaSource interface {
|
||||
@@ -616,4 +657,61 @@ func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor
|
||||
for _, reason := range s.Throttle {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
|
||||
}
|
||||
if w := s.Wear; w != nil {
|
||||
c.gpuWear(ch, w, labels)
|
||||
}
|
||||
}
|
||||
|
||||
// gpuWear is what wears a card out. A count the card does not keep (-1) is
|
||||
// left out, not published as zero.
|
||||
func (c *Collector) gpuWear(ch chan<- prometheus.Metric, w *inventory.GPUWear, labels []string) {
|
||||
gauge := func(d *prometheus.Desc, v float64, extra ...string) {
|
||||
ch <- prometheus.MustNewConstMetric(d, prometheus.GaugeValue, v, append(append([]string{}, labels...), extra...)...)
|
||||
}
|
||||
flag := func(b bool) float64 {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
if w.PCIeReplays >= 0 {
|
||||
gauge(c.gpuPCIeReplay, float64(w.PCIeReplays))
|
||||
}
|
||||
if w.RetiredSBE >= 0 || w.RetiredDBE >= 0 {
|
||||
gauge(c.gpuRetired, float64(max(w.RetiredSBE, 0)), "sbe")
|
||||
gauge(c.gpuRetired, float64(max(w.RetiredDBE, 0)), "dbe")
|
||||
gauge(c.gpuRetirePend, flag(w.RetirePending))
|
||||
}
|
||||
if w.RemappedCorrectable >= 0 || w.RemappedUncorrectable >= 0 {
|
||||
gauge(c.gpuRemapped, float64(max(w.RemappedCorrectable, 0)), "correctable")
|
||||
gauge(c.gpuRemapped, float64(max(w.RemappedUncorrectable, 0)), "uncorrectable")
|
||||
gauge(c.gpuRemapPend, flag(w.RemapPending))
|
||||
gauge(c.gpuRemapFail, flag(w.RemapFailure))
|
||||
}
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkCRC), "crc")
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkReplay), "replay")
|
||||
gauge(c.gpuNVLinkErr, float64(w.NVLinkRecovery), "recovery")
|
||||
gauge(c.gpuXID, float64(w.XIDs))
|
||||
if w.XIDs > 0 {
|
||||
gauge(c.gpuLastXID, float64(w.LastXID))
|
||||
}
|
||||
}
|
||||
|
||||
// collectAssignments says which replica holds each GPU.
|
||||
func (c *Collector) collectAssignments(ch chan<- prometheus.Metric) {
|
||||
var owners map[string]string
|
||||
if o, ok := c.src.(OwnerSource); ok {
|
||||
owners = o.ModelOwners()
|
||||
}
|
||||
for _, r := range c.src.Replicas() {
|
||||
if !r.State.Counts() {
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
for _, g := range sh.GPUIndices {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuAssigned, prometheus.GaugeValue, 1,
|
||||
sh.NodeID, strconv.Itoa(g), r.Model, r.ID, owners[r.Model])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
@@ -77,3 +78,52 @@ func TestHowBusyAGPUIsComesFromTheLatestSampleNotRegistration(t *testing.T) {
|
||||
t.Errorf("a utilization the driver did not report was published:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// ownedSource adds owners to fakeSource.
|
||||
type ownedSource struct{ fakeSource }
|
||||
|
||||
func (ownedSource) ModelOwners() map[string]string { return map[string]string{"chat": "acme/ml"} }
|
||||
|
||||
func TestEachGPUAReplicaHoldsSaysWhoseItIs(t *testing.T) {
|
||||
src := ownedSource{fakeSource{replicas: []reconciler.Replica{
|
||||
{ID: "chat-1", Model: "chat", State: reconciler.Ready, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{0, 1}}}},
|
||||
{ID: "old-1", Model: "old", State: reconciler.Stopped, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}},
|
||||
}}}
|
||||
out := gather(t, src)
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_assigned gpu=0 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
|
||||
"llmgw_gpu_assigned gpu=1 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s in:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "replica=old-1") {
|
||||
t.Error("a stopped replica's GPU was counted as held")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGPUsWearIsPublishedAndWhatItDoesNotKeepIsLeftOut(t *testing.T) {
|
||||
n := inventory.Node{ID: "g1", State: inventory.Ready,
|
||||
GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559}},
|
||||
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 40, MemoryTotalMB: 81559,
|
||||
Wear: &inventory.GPUWear{PCIeReplays: 12, RetiredSBE: -1, RetiredDBE: -1, RemappedCorrectable: 2,
|
||||
RemappedUncorrectable: 1, RemapPending: true, NVLinkCRC: 7, XIDs: 3, LastXID: 79}}}}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_pcie_replays class=h100-80g gpu=0 node=g1 -> 12\n",
|
||||
"llmgw_gpu_remapped_rows class=h100-80g gpu=0 kind=uncorrectable node=g1 -> 1\n",
|
||||
"llmgw_gpu_remap_pending class=h100-80g gpu=0 node=g1 -> 1\n",
|
||||
"llmgw_gpu_nvlink_errors class=h100-80g gpu=0 kind=crc node=g1 -> 7\n",
|
||||
"llmgw_gpu_xid_errors class=h100-80g gpu=0 node=g1 -> 3\n",
|
||||
"llmgw_gpu_last_xid class=h100-80g gpu=0 node=g1 -> 79\n",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s", want)
|
||||
}
|
||||
}
|
||||
// An H100 keeps no retired pages; none are published.
|
||||
if strings.Contains(out, "llmgw_gpu_retired_pages") {
|
||||
t.Errorf("a count the card does not keep was published:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@
|
||||
package inventory
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -328,6 +329,60 @@ type GPUSensor struct {
|
||||
MemoryTotalMB float64 `json:"memory_total_mb,omitempty"`
|
||||
SMClockMHz float64 `json:"sm_clock_mhz,omitempty"`
|
||||
ECCUncorrected float64 `json:"ecc_uncorrected,omitempty"`
|
||||
// Wear is what wears the card out, read every minute; nil from an agent
|
||||
// that does not read it.
|
||||
Wear *GPUWear `json:"wear,omitempty"`
|
||||
}
|
||||
|
||||
// GPUWear is a card's wear and failures. Counts of -1 are not reported for
|
||||
// it: a card before Ampere has no remapped rows, one after no retired pages.
|
||||
type GPUWear struct {
|
||||
PCIeReplays int64 `json:"pcie_replays"`
|
||||
RetiredSBE int64 `json:"retired_pages_sbe"`
|
||||
RetiredDBE int64 `json:"retired_pages_dbe"`
|
||||
RetirePending bool `json:"retire_pending,omitempty"`
|
||||
RemappedCorrectable int64 `json:"remapped_rows_correctable"`
|
||||
RemappedUncorrectable int64 `json:"remapped_rows_uncorrectable"`
|
||||
RemapPending bool `json:"remap_pending,omitempty"`
|
||||
RemapFailure bool `json:"remap_failure,omitempty"`
|
||||
NVLinkCRC int64 `json:"nvlink_crc_errors"`
|
||||
NVLinkReplay int64 `json:"nvlink_replay_errors"`
|
||||
NVLinkRecovery int64 `json:"nvlink_recovery_errors"`
|
||||
XIDs int64 `json:"xid_errors"`
|
||||
LastXID int `json:"last_xid,omitempty"`
|
||||
LastXIDText string `json:"last_xid_text,omitempty"`
|
||||
LastXIDAt time.Time `json:"last_xid_at,omitempty"`
|
||||
}
|
||||
|
||||
// Problems says, in a few words each, what is wrong with the card; empty
|
||||
// when nothing is.
|
||||
func (w *GPUWear) Problems() []string {
|
||||
if w == nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
if w.RemapFailure {
|
||||
out = append(out, "out of spare memory rows: replace the card")
|
||||
}
|
||||
if w.RemapPending {
|
||||
out = append(out, "memory rows remapped, waiting for a reset")
|
||||
}
|
||||
if w.RetirePending {
|
||||
out = append(out, "memory pages waiting for a reset to be retired")
|
||||
}
|
||||
if w.RemappedUncorrectable > 0 {
|
||||
out = append(out, fmt.Sprintf("%d rows remapped for uncorrectable errors", w.RemappedUncorrectable))
|
||||
}
|
||||
if w.RetiredDBE > 0 {
|
||||
out = append(out, fmt.Sprintf("%d pages retired for double-bit errors", w.RetiredDBE))
|
||||
}
|
||||
if w.XIDs > 0 {
|
||||
out = append(out, fmt.Sprintf("XID %d (%s)", w.LastXID, w.LastXIDText))
|
||||
}
|
||||
if w.NVLinkCRC > 0 || w.NVLinkReplay > 0 || w.NVLinkRecovery > 0 {
|
||||
out = append(out, fmt.Sprintf("NVLink errors: %d CRC, %d replay, %d recovery", w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Busy reports whether the sample carries how busy the card is.
|
||||
|
||||
+13
-1
@@ -255,6 +255,18 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
}
|
||||
seen := map[string]struct{}{}
|
||||
var out []web.ChatModel
|
||||
// What each model's replicas compute on, across all of them: one model
|
||||
// can run on a card and on CPUs at once.
|
||||
hardware := map[string][]string{}
|
||||
for _, s := range r.reg.Snapshots("") {
|
||||
hw := hardwareOf(s.Pool)
|
||||
for _, m := range s.Models {
|
||||
name, _ := r.resolve(m)
|
||||
if hw != "" && !slices.Contains(hardware[name], hw) {
|
||||
hardware[name] = append(hardware[name], hw)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, s := range r.reg.Snapshots("") {
|
||||
for _, m := range s.Models {
|
||||
// Resolved rather than taken as sent: an alias and the name it
|
||||
@@ -265,7 +277,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated()})
|
||||
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -203,6 +203,9 @@ func (r *Router) Handler() http.Handler {
|
||||
}
|
||||
pages.RegisterAdminQuotas(mux, r.store, r.acct)
|
||||
// And the report of what that tree actually used.
|
||||
if r.history != nil {
|
||||
pages.SetHistory(r.history)
|
||||
}
|
||||
pages.RegisterAdminUsage(mux, r.store, r.acct)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
|
||||
func TestAReplySaysWhatHardwareAnsweredIt(t *testing.T) {
|
||||
// qwen2.5-coder-14b had one replica on cuda01's card and one on a CPU
|
||||
// node, and the chat page could not tell which had answered: the same
|
||||
// model, ten times faster or slower, for no visible reason.
|
||||
for pool, want := range map[string]string{
|
||||
"none/6000-24g": "6000-24g",
|
||||
"none/cpu": "cpu",
|
||||
"": "",
|
||||
} {
|
||||
e := newFakeEngine(t, "m")
|
||||
rt := newTestRouter(t, baseConfig())
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "r", URL: e.srv.URL, Models: []string{"m"}, Pool: pool, Weight: 1},
|
||||
})
|
||||
waitForEndpoint(t, rt)
|
||||
|
||||
rec := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("%q: status %d: %s", pool, rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := rec.Header().Get(hardwareHeader); got != want {
|
||||
t.Errorf("pool %q: hardware %q, want %q", pool, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestThePickerSaysWhereEachModelRuns(t *testing.T) {
|
||||
gpu, cpu := newFakeEngine(t, "qwen2.5-coder-7b-instruct"), newFakeEngine(t, "qwen2.5-coder-7b-instruct")
|
||||
rt, user := chatRouter(t, gpu.srv.URL)
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "g", URL: gpu.srv.URL, Models: []string{"qwen2.5-coder-7b-instruct"}, Pool: "none/6000-24g", Weight: 1},
|
||||
{ID: "c", URL: cpu.srv.URL, Models: []string{"coder"}, Pool: "none/cpu", Weight: 1},
|
||||
})
|
||||
waitForEndpoint(t, rt)
|
||||
|
||||
got := rt.modelsFor(httptest.NewRequest("GET", "/chat", nil), user)
|
||||
if len(got) != 1 || got[0].Runs() != "GPU + CPU" {
|
||||
t.Fatalf("picker %+v; want one model running on GPU + CPU, its alias's replica included", got)
|
||||
}
|
||||
}
|
||||
@@ -749,6 +749,16 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
if a, ok := r.adapter(req.Model); ok && a.base == model {
|
||||
body = withAdapter(body, model, a.index, a.scale)
|
||||
}
|
||||
// What computes the answer, said before the proxy writes the headers.
|
||||
// A model can have replicas on cards and on CPUs at once, and the two
|
||||
// differ tenfold in speed: a reply that came slowly is otherwise
|
||||
// indistinguishable from a model that is slow everywhere. Set again on
|
||||
// a retry, which only happens before anything was written.
|
||||
if hw := hardwareOf(dec.Endpoint.Pool); hw != "" {
|
||||
w.Header().Set(hardwareHeader, hw)
|
||||
} else {
|
||||
w.Header().Del(hardwareHeader)
|
||||
}
|
||||
out, perr := r.px.Do(uctx, w, proxy.Request{
|
||||
URL: dec.Endpoint.URL + req.Path,
|
||||
APIKey: dec.Endpoint.APIKey,
|
||||
@@ -791,6 +801,21 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
return last
|
||||
}
|
||||
|
||||
// hardwareHeader says what the replica that answered computes on: "cpu", or
|
||||
// the GPU class, such as "6000-24g".
|
||||
const hardwareHeader = "x-llmgw-hardware"
|
||||
|
||||
// hardwareOf reads it from a pool, which placement names fabric/class —
|
||||
// "none/cpu", "none/6000-24g", "ib-a/h100-80g". Empty for an endpoint
|
||||
// configured by hand, which has no pool and says nothing about its hardware.
|
||||
func hardwareOf(pool string) string {
|
||||
i := strings.LastIndexByte(pool, '/')
|
||||
if i < 0 || i == len(pool)-1 {
|
||||
return ""
|
||||
}
|
||||
return pool[i+1:]
|
||||
}
|
||||
|
||||
func without(eps []endpoint.Snapshot, id string) []endpoint.Snapshot {
|
||||
return slices.DeleteFunc(slices.Clone(eps), func(e endpoint.Snapshot) bool { return e.ID == id })
|
||||
}
|
||||
|
||||
@@ -64,6 +64,9 @@ type AccountPages struct {
|
||||
// hasQuotas is set when /admin/quotas is registered.
|
||||
hasQuotas bool
|
||||
// capacity sets capacity quotas, from the quotas page; nil leaves them off.
|
||||
// history is the metrics store, for what only it knows: the GPU time
|
||||
// each owner held. Nil without one.
|
||||
history History
|
||||
capacity CapacityAdmin
|
||||
// personalOff refuses personal keys; see SetPersonalOff.
|
||||
personalOff bool
|
||||
@@ -85,6 +88,13 @@ type AccountPages struct {
|
||||
// could run it. Both nil where the mirror page is read-only.
|
||||
catalog CatalogAdmin
|
||||
fleet Source
|
||||
// The stores behind the people page's three tabs — the tree and its
|
||||
// people, the limits, the usage — each set when its part is mounted, so
|
||||
// the page draws the parts that exist.
|
||||
peopleStore AccountAdmin
|
||||
quotaStore QuotaAdmin
|
||||
usageStore UsageAdmin
|
||||
usagePriced Priced
|
||||
}
|
||||
|
||||
// ServedModels supplies the names the profile page's example should use.
|
||||
@@ -651,3 +661,6 @@ func errorRef() string {
|
||||
_, _ = rand.Read(b[:])
|
||||
return "err-" + hex.EncodeToString(b[:])
|
||||
}
|
||||
|
||||
// SetHistory gives the pages the metrics store.
|
||||
func (p *AccountPages) SetHistory(h History) { p.history = h }
|
||||
@@ -0,0 +1,259 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// The people page's tabs. One page, because people, the limits they are held
|
||||
// to and what they used are three views of one tree: managing a team meant its
|
||||
// members on one page, its quota on another and its spend on a third, each
|
||||
// linking to the others. Each tab keeps the address it had as a page of its
|
||||
// own, so links and bookmarks to /admin/quotas and /admin/usage still land.
|
||||
const (
|
||||
tabPeople = "people"
|
||||
tabLimits = "limits"
|
||||
tabUsage = "usage"
|
||||
)
|
||||
|
||||
// peopleView is what a request asks the people page for.
|
||||
type peopleView struct {
|
||||
Tab string
|
||||
Msg, Err string
|
||||
// Open is the project shown expanded in the tree.
|
||||
Open string
|
||||
}
|
||||
|
||||
type peopleData struct {
|
||||
Nav Nav
|
||||
Tab string
|
||||
// The tabs this server has, each built only when it is the one shown.
|
||||
HasPeople, HasLimits, HasUsage bool
|
||||
Users *adminUsersData
|
||||
Quotas *adminQuotasData
|
||||
Usage *adminUsageData
|
||||
// Detail is the branch or person chosen in the tree, nil when none is.
|
||||
Detail *peopleDetail
|
||||
// Sel is what was chosen, "team:<id>" or "user:<id>", for links to keep.
|
||||
Sel string
|
||||
Pending int
|
||||
Version string
|
||||
Now string
|
||||
}
|
||||
|
||||
// peopleDetail is one organization, team, project or person, with everything
|
||||
// the three tabs say about it in one place.
|
||||
type peopleDetail struct {
|
||||
Kind store.Scope
|
||||
ID string
|
||||
// Name is what it is called, Path where it sits.
|
||||
Name, Path string
|
||||
|
||||
// Limits is its own quota row, with what it is held to from above; nil
|
||||
// for a person, or where limits are not mounted.
|
||||
Limits *quotaRow
|
||||
// Person is a person's own limit.
|
||||
Person *personRow
|
||||
// Memberships are where a person is.
|
||||
Memberships []store.Membership
|
||||
// Capacity is its capacity quota, nil when it has none.
|
||||
Capacity *capacityRow
|
||||
HasCapacity bool
|
||||
BudgetsBind bool
|
||||
|
||||
// Its use over the last week, and the totals beside it.
|
||||
HasUsage bool
|
||||
Requests, Tokens, Spend TimeSeries
|
||||
Totals usageRow
|
||||
Priced bool
|
||||
UsageWindow, QuotaWindow string
|
||||
|
||||
CSRF string
|
||||
}
|
||||
|
||||
// renderPeople draws the people page on the tab asked for.
|
||||
func (p *AccountPages) renderPeople(w http.ResponseWriter, r *http.Request, admin store.User, v peopleView) {
|
||||
d := peopleData{
|
||||
Nav: navFrom(r, "people"), Tab: v.Tab,
|
||||
HasPeople: p.peopleStore != nil, HasLimits: p.quotaStore != nil, HasUsage: p.usageStore != nil,
|
||||
Sel: r.FormValue("sel"),
|
||||
Version: p.version, Now: time.Now().UTC().Format(time.RFC3339),
|
||||
}
|
||||
switch {
|
||||
case d.Tab == tabPeople && d.HasPeople:
|
||||
u := p.buildUsers(r, p.peopleStore, admin, v.Msg, v.Err, v.Open)
|
||||
d.Users, d.Pending = &u, len(u.Pending)
|
||||
if d.Sel != "" {
|
||||
d.Detail = p.detail(r, d.Sel, &u)
|
||||
}
|
||||
case d.Tab == tabLimits && d.HasLimits:
|
||||
q := p.buildQuotas(r, p.quotaStore, v.Msg, v.Err)
|
||||
d.Quotas = &q
|
||||
case d.Tab == tabUsage && d.HasUsage:
|
||||
u := p.buildUsage(r, p.usageStore, p.usagePriced)
|
||||
d.Usage = &u
|
||||
default:
|
||||
// The tab asked for is not mounted here: the first one that is.
|
||||
switch {
|
||||
case d.HasPeople:
|
||||
v.Tab = tabPeople
|
||||
case d.HasLimits:
|
||||
v.Tab = tabLimits
|
||||
case d.HasUsage:
|
||||
v.Tab = tabUsage
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
p.renderPeople(w, r, admin, v)
|
||||
return
|
||||
}
|
||||
render(w, "admin_people", d)
|
||||
}
|
||||
|
||||
// usageGrouping is the grouping a branch's or a person's use is read under.
|
||||
func usageGrouping(kind store.Scope) (store.Grouping, bool) {
|
||||
switch kind {
|
||||
case store.ScopeOrg:
|
||||
return store.GroupByOrg, true
|
||||
case store.ScopeTeam:
|
||||
return store.GroupByTeam, true
|
||||
case store.ScopeProject:
|
||||
return store.GroupByProject, true
|
||||
case store.ScopeUser:
|
||||
return store.GroupByUser, true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// detail gathers what the page knows about one branch or person.
|
||||
func (p *AccountPages) detail(r *http.Request, sel string, u *adminUsersData) *peopleDetail {
|
||||
kind, id, ok := strings.Cut(sel, ":")
|
||||
if !ok || id == "" {
|
||||
return nil
|
||||
}
|
||||
ctx := r.Context()
|
||||
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfFor(r),
|
||||
UsageWindow: "last 7 days", QuotaWindow: "30 days"}
|
||||
if !p.nameDetail(d, u) {
|
||||
return nil
|
||||
}
|
||||
|
||||
if p.quotaStore != nil {
|
||||
d.BudgetsBind = p.priced != nil && p.priced.AnythingPriced()
|
||||
p.limitsDetail(ctx, d)
|
||||
}
|
||||
if p.capacity != nil && d.Kind != store.ScopeUser {
|
||||
d.HasCapacity = true
|
||||
if views, err := p.capacity.QuotaViews(ctx); err == nil {
|
||||
for _, v := range views {
|
||||
if v.Scope == sel {
|
||||
c := capacityRow{QuotaView: v,
|
||||
GPUs: capCell(v.Holds.GPUs, v.Guarantee.GPUs, v.Limit.GPUs),
|
||||
Cores: capCell(v.Holds.CPUCores, v.Guarantee.CPUCores, v.Limit.CPUCores),
|
||||
Memory: capCell(v.Holds.MemoryGB, v.Guarantee.MemoryGB, v.Limit.MemoryGB)}
|
||||
d.Capacity = &c
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if p.usageStore != nil {
|
||||
p.usageDetail(ctx, d)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// nameDetail finds what was chosen in the tree or among the accounts, and
|
||||
// says whether it is there at all.
|
||||
func (p *AccountPages) nameDetail(d *peopleDetail, u *adminUsersData) bool {
|
||||
if d.Kind == store.ScopeUser {
|
||||
for _, a := range u.Accounts {
|
||||
if a.ID == d.ID {
|
||||
d.Name, d.Path, d.Memberships = a.Email, a.Name, a.Memberships
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
for _, o := range u.Tree {
|
||||
if d.Kind == store.ScopeOrg && o.ID == d.ID {
|
||||
d.Name, d.Path = o.Name, o.Name
|
||||
return true
|
||||
}
|
||||
for _, t := range o.Teams {
|
||||
if d.Kind == store.ScopeTeam && t.ID == d.ID {
|
||||
d.Name, d.Path = t.Name, o.Name+" / "+t.Name
|
||||
return true
|
||||
}
|
||||
for _, pr := range t.Projects {
|
||||
if d.Kind == store.ScopeProject && pr.ID == d.ID {
|
||||
d.Name, d.Path = pr.Name, o.Name+" / "+t.Name+" / "+pr.Name
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// limitsDetail is the branch's own quota row, or the person's own limit.
|
||||
func (p *AccountPages) limitsDetail(ctx context.Context, d *peopleDetail) {
|
||||
s := p.quotaStore
|
||||
limits, err := s.AllLimits(ctx)
|
||||
if err != nil {
|
||||
p.log.Warn("could not read the limits for a detail panel", "err", err)
|
||||
return
|
||||
}
|
||||
since := time.Now().Add(-quotaWindow)
|
||||
if d.Kind == store.ScopeUser {
|
||||
for _, row := range p.personRows(ctx, s, limits, since, "") {
|
||||
if row.ID == d.ID {
|
||||
row := row
|
||||
d.Person = &row
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
tree, err := s.Tree(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
for _, row := range quotaRows(tree, limits, p.spendByScope(ctx, s, since)) {
|
||||
if row.Scope == d.Kind && row.ID == d.ID {
|
||||
ms := store.MemberScope(row.Scope)
|
||||
if l, ok := limits[store.LimitOwner{Scope: ms, ID: row.ID}.CounterKey()]; ok {
|
||||
row.MemberRaw, row.Member = fieldsOf(l), describeLimits(l)
|
||||
}
|
||||
d.Limits = &row
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// usageDetail is the branch's or person's use over the last week, drawn.
|
||||
func (p *AccountPages) usageDetail(ctx context.Context, d *peopleDetail) {
|
||||
g, ok := usageGrouping(d.Kind)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
win := windowFor("7d")
|
||||
rows, err := p.usageStore.UsageBucketsBy(ctx, g, time.Now().Add(-win.Span), win.Width)
|
||||
if err != nil {
|
||||
p.log.Warn("could not read usage for a detail panel", "sel", string(d.Kind)+":"+d.ID, "err", err)
|
||||
return
|
||||
}
|
||||
d.HasUsage = true
|
||||
d.Priced = p.usagePriced != nil && p.usagePriced.AnythingPriced()
|
||||
for _, s := range rows {
|
||||
if s.ID == d.ID {
|
||||
var table []usageRow
|
||||
d.Requests, d.Tokens, d.Spend, table = usageCharts([]store.UsageSeries{s})
|
||||
d.Totals = table[0]
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// peoplePage mounts all three tabs, as the server does.
|
||||
func peoplePage(t *testing.T, q *fakeQuotas, u *fakeScopeSeries) *http.ServeMux {
|
||||
t.Helper()
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsers(mux, &fakeAdmin{orgs: q.orgs})
|
||||
p.RegisterAdminQuotas(mux, q, pricedYes{})
|
||||
p.RegisterAdminUsage(mux, u, pricedYes{})
|
||||
return mux
|
||||
}
|
||||
|
||||
func TestABranchChosenInTheTreeShowsItsLimitsAndUseTogether(t *testing.T) {
|
||||
// People, their limits and their use were three pages: a team's members
|
||||
// on one, its quota on another, its spend on a third.
|
||||
q := oneTeam()
|
||||
rpm := 60
|
||||
q.limits["org:org-1"] = store.Limits{Scope: store.ScopeOrg, ScopeID: "org-1", RPM: &rpm}
|
||||
u := &fakeScopeSeries{byGroup: map[store.Grouping][]store.UsageSeries{
|
||||
store.GroupByOrg: {{Grouping: store.GroupByOrg, ID: "org-1", Name: "Scania", Requests: 1234,
|
||||
Buckets: []store.Bucket{{Start: time.Now().Add(-time.Hour), Requests: 1234}}}},
|
||||
}}
|
||||
body := strings.Join(strings.Fields(
|
||||
mirrorGet(t, peoplePage(t, q, u), "/admin/users?sel=org:org-1").Body.String()), " ")
|
||||
|
||||
for _, want := range []string{
|
||||
`class="people-detail"`, "<h2>Scania</h2>",
|
||||
"<dt>Requests/min</dt><dd>60</dd>", // its own limit
|
||||
"1.2k requests", // its use over the week
|
||||
`name="sel" value="org:org-1"`, // changing it comes back here
|
||||
`aria-current="page">Organizations & people`,
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the panel lacks %q", want)
|
||||
}
|
||||
}
|
||||
if u.asked[len(u.asked)-1] != store.GroupByOrg {
|
||||
t.Errorf("use was read by %v, not by organization", u.asked)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAChangeMadeInAPanelComesBackToThePanel(t *testing.T) {
|
||||
q := oneTeam()
|
||||
body := mirrorPost(t, peoplePage(t, q, &fakeScopeSeries{}), "/admin/quotas", url.Values{
|
||||
"csrf": {"tok"}, "scope": {"org"}, "id": {"org-1"}, "rpm": {"30"},
|
||||
"tab": {"people"}, "sel": {"org:org-1"},
|
||||
}).Body.String()
|
||||
if len(q.put) != 1 || *q.put[0].RPM != 30 {
|
||||
t.Fatalf("the limit was not saved: %+v", q.put)
|
||||
}
|
||||
if !strings.Contains(body, `class="people-detail"`) || !strings.Contains(body, "rpm 30") {
|
||||
t.Error("saving from the panel did not come back to it with the result")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachOldAddressOpensItsTab(t *testing.T) {
|
||||
mux := peoplePage(t, oneTeam(), &fakeScopeSeries{})
|
||||
for path, tab := range map[string]string{
|
||||
"/admin/users": "Organizations & people", "/admin/quotas": "Limits", "/admin/usage": "Usage",
|
||||
} {
|
||||
body := mirrorGet(t, mux, path).Body.String()
|
||||
if !strings.Contains(body, `aria-current="page">`+tab) {
|
||||
t.Errorf("%s does not open the %s tab", path, tab)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSomethingNotInTheTreeOpensNoPanel(t *testing.T) {
|
||||
body := mirrorGet(t, peoplePage(t, oneTeam(), &fakeScopeSeries{}), "/admin/users?sel=org:gone").Body.String()
|
||||
if strings.Contains(body, `class="people-detail"`) {
|
||||
t.Error("a panel opened for an organization that does not exist")
|
||||
}
|
||||
}
|
||||
@@ -159,10 +159,11 @@ type adminQuotasData struct {
|
||||
func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, priced Priced) {
|
||||
p.hasQuotas = true
|
||||
p.priced = priced
|
||||
p.quotaStore = s
|
||||
|
||||
mux.HandleFunc("GET /admin/quotas", p.adminOnly(
|
||||
func(w http.ResponseWriter, r *http.Request, _ store.User) {
|
||||
p.renderQuotas(w, r, s, "", "")
|
||||
func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
p.renderPeople(w, r, admin, peopleView{Tab: tabLimits})
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/quotas", p.adminOnly(
|
||||
@@ -177,10 +178,10 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
|
||||
p.log.Warn("a quota change was refused",
|
||||
"admin", admin.Email, "scope", r.FormValue("scope"),
|
||||
"id", r.FormValue("id"), "err", err)
|
||||
p.renderQuotas(w, r, s, "", err.Error())
|
||||
p.renderQuotas(w, r, admin, "", err.Error())
|
||||
return
|
||||
}
|
||||
p.renderQuotas(w, r, s, msg, "")
|
||||
p.renderQuotas(w, r, admin, msg, "")
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/quotas/capacity", p.adminOnly(
|
||||
@@ -194,11 +195,11 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
|
||||
if err != nil {
|
||||
p.log.Warn("a capacity quota change was refused",
|
||||
"admin", admin.Email, "scope", r.FormValue("scope"), "err", err)
|
||||
p.renderQuotas(w, r, s, "", err.Error())
|
||||
p.renderQuotas(w, r, admin, "", err.Error())
|
||||
return
|
||||
}
|
||||
p.log.Info("capacity quota changed", "admin", admin.Email, "what", msg)
|
||||
p.renderQuotas(w, r, s, msg, "")
|
||||
p.renderQuotas(w, r, admin, msg, "")
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -270,10 +271,24 @@ func (p *AccountPages) applyQuota(r *http.Request, s QuotaAdmin, admin store.Use
|
||||
return string(scope) + " " + id + ": " + now, nil
|
||||
}
|
||||
|
||||
// renderQuotas answers a change to a limit: on the limits tab, or on the
|
||||
// people tab when it was made from a branch's own panel there.
|
||||
func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
|
||||
s QuotaAdmin, msg, errMsg string,
|
||||
admin store.User, msg, errMsg string,
|
||||
) {
|
||||
const window = 30 * 24 * time.Hour
|
||||
tab := tabLimits
|
||||
if r.FormValue("tab") == tabPeople {
|
||||
tab = tabPeople
|
||||
}
|
||||
p.renderPeople(w, r, admin, peopleView{Tab: tab, Msg: msg, Err: errMsg})
|
||||
}
|
||||
|
||||
// quotaWindow is how far back spend against a budget is counted.
|
||||
const quotaWindow = 30 * 24 * time.Hour
|
||||
|
||||
// buildQuotas is every limit, as the limits tab shows them.
|
||||
func (p *AccountPages) buildQuotas(r *http.Request, s QuotaAdmin, msg, errMsg string) adminQuotasData {
|
||||
const window = quotaWindow
|
||||
d := adminQuotasData{
|
||||
Nav: navFrom(r, "quotas"),
|
||||
Since: "30 days", Message: msg, Error: errMsg,
|
||||
@@ -289,8 +304,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
|
||||
if err != nil {
|
||||
p.log.Error("could not read the limits", "err", err)
|
||||
d.Error = "could not read the limits"
|
||||
render(w, "admin_quotas", d)
|
||||
return
|
||||
return d
|
||||
}
|
||||
spend := p.spendByScope(r.Context(), s, time.Now().Add(-window))
|
||||
|
||||
@@ -314,7 +328,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
|
||||
d.Rows[i].Editing = d.Editing ==
|
||||
string(d.Rows[i].Scope)+":"+d.Rows[i].ID
|
||||
}
|
||||
render(w, "admin_quotas", d)
|
||||
return d
|
||||
}
|
||||
|
||||
// spendByScope is what each group has used in the window, keyed like a limit.
|
||||
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
@@ -80,6 +81,12 @@ type adminUsageData struct {
|
||||
// chart along the bottom reads as "nobody spent anything".
|
||||
Priced bool
|
||||
|
||||
// GPUHeld and GPUHours are the cards each owner's models held over the
|
||||
// window, from the metrics store; empty without one, or with no owned
|
||||
// model on a GPU.
|
||||
GPUHeld TimeSeries
|
||||
GPUHours []gpuHoursRow
|
||||
|
||||
Error string
|
||||
Version string
|
||||
Now string
|
||||
@@ -92,15 +99,15 @@ type adminUsageData struct {
|
||||
// answers "who is using the cluster, and when" — including per person, which is
|
||||
// not a level quotas can be written on at all.
|
||||
func (p *AccountPages) RegisterAdminUsage(mux *http.ServeMux, s UsageAdmin, priced Priced) {
|
||||
mux.HandleFunc("GET /admin/usage", p.AdminPage(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
p.renderUsage(w, r, s, priced)
|
||||
p.usageStore, p.usagePriced = s, priced
|
||||
mux.HandleFunc("GET /admin/usage", p.adminOnly(
|
||||
func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
p.renderPeople(w, r, admin, peopleView{Tab: tabUsage})
|
||||
}))
|
||||
}
|
||||
|
||||
func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
|
||||
s UsageAdmin, priced Priced,
|
||||
) {
|
||||
// buildUsage is the cluster's use over time, as the usage tab shows it.
|
||||
func (p *AccountPages) buildUsage(r *http.Request, s UsageAdmin, priced Priced) adminUsageData {
|
||||
by := store.Grouping(r.URL.Query().Get("by"))
|
||||
if !by.Valid() {
|
||||
by = store.GroupByTeam
|
||||
@@ -130,12 +137,48 @@ func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
|
||||
if err != nil {
|
||||
p.log.Error("could not read usage over time", "by", by, "err", err)
|
||||
d.Error = "could not read usage"
|
||||
render(w, "admin_usage", d)
|
||||
return
|
||||
return d
|
||||
}
|
||||
|
||||
d.Requests, d.Tokens, d.Spend, d.Rows = usageCharts(rows)
|
||||
render(w, "admin_usage", d)
|
||||
d.GPUHeld, d.GPUHours = p.gpuHours(r.Context(), win)
|
||||
return d
|
||||
}
|
||||
|
||||
// gpuHoursRow is the GPU time one owner's models held over the window.
|
||||
type gpuHoursRow struct {
|
||||
Owner string
|
||||
Hours float64
|
||||
// Now is how many cards they hold at the end of the window.
|
||||
Now float64
|
||||
}
|
||||
|
||||
// gpuHours is the cards held by each owner's models over the window, drawn
|
||||
// and summed into GPU-hours. From the metrics store, because what a model
|
||||
// holds is placement's, not the request log's: a model nobody asks still
|
||||
// holds its cards. As far back as the store keeps history, which for a
|
||||
// default Prometheus is fifteen days.
|
||||
func (p *AccountPages) gpuHours(ctx context.Context, win usageWindow) (TimeSeries, []gpuHoursRow) {
|
||||
if p.history == nil {
|
||||
return TimeSeries{}, nil
|
||||
}
|
||||
series, err := p.history.RangeQuery(ctx, `sum by (owner) (llmgw_gpu_assigned)`, "owner", win.Span, win.Width)
|
||||
if err != nil || len(series) == 0 {
|
||||
return TimeSeries{}, nil
|
||||
}
|
||||
var rows []gpuHoursRow
|
||||
for i := range series {
|
||||
if series[i].Name == "" {
|
||||
series[i].Name = "no owner"
|
||||
}
|
||||
row := gpuHoursRow{Owner: series[i].Name, Now: series[i].Last()}
|
||||
// Each point is the cards held at that instant, standing for the
|
||||
// step that follows it.
|
||||
row.Hours = series[i].Sum() * win.Width.Hours()
|
||||
rows = append(rows, row)
|
||||
}
|
||||
sort.Slice(rows, func(i, j int) bool { return rows[i].Hours > rows[j].Hours })
|
||||
return LineChart(series, "", 6), rows
|
||||
}
|
||||
|
||||
// seriesOf turns each group's buckets into a line of one measure.
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -162,3 +163,32 @@ func TestAPersonWithNoAccountIsNamedRatherThanBlank(t *testing.T) {
|
||||
t.Errorf("unowned traffic was left unlabelled; got %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheUsagePageCountsTheGPUTimeEachOwnerHeld(t *testing.T) {
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
// Over a day in 20-minute steps: team:research held 8 cards throughout,
|
||||
// nobody's model 2 for the last three steps.
|
||||
res, none := promq.Series{Name: "team:research"}, promq.Series{Name: ""}
|
||||
for i := range 72 {
|
||||
res.Points = append(res.Points, promq.Point{Value: 8})
|
||||
if i >= 69 {
|
||||
none.Points = append(none.Points, promq.Point{Value: 2})
|
||||
}
|
||||
}
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{`sum by (owner) (llmgw_gpu_assigned)`: {res, none}}}
|
||||
p.SetHistory(h)
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsage(mux, &fakeScopeSeries{}, nil)
|
||||
body := mirrorGet(t, mux, "/admin/usage?window=24h").Body.String()
|
||||
for _, want := range []string{"GPU time, by owner", "<td>team:research</td><td class=\"num\">192.0</td>",
|
||||
"<td>no owner</td><td class=\"num\">2.0</td>"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// Without a metrics store, nothing is said about GPU time.
|
||||
if body := mirrorGet(t, usagePage(t, &fakeScopeSeries{}, nil), "/admin/usage").Body.String(); strings.Contains(body, "GPU time") {
|
||||
t.Error("GPU time shown with no metrics store to read it from")
|
||||
}
|
||||
}
|
||||
@@ -87,6 +87,9 @@ type adminUsersData struct {
|
||||
Projects []store.ProjectPath
|
||||
Orgs []store.Org
|
||||
Teams []store.TeamPath
|
||||
// Sel is the branch or person whose panel is open, "team:<id>", so the
|
||||
// tree can mark it.
|
||||
Sel string
|
||||
// Open is the project whose section is shown expanded: the one just acted
|
||||
// on, so the result of a change is in view without the reader hunting for
|
||||
// it. Scripted pages keep every open section open; this is for the rest.
|
||||
@@ -115,6 +118,7 @@ type adminUsersData struct {
|
||||
// shell is no hardship. Revoking a key is here, because it only ever takes
|
||||
// access away.
|
||||
func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
|
||||
p.peopleStore = s
|
||||
mux.HandleFunc("GET /admin/users", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
p.renderUsers(w, r, s, admin, "", "", r.URL.Query().Get("open"))
|
||||
}))
|
||||
@@ -480,12 +484,17 @@ func (p *AccountPages) adminAction(s AccountAdmin,
|
||||
}
|
||||
|
||||
func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
|
||||
s AccountAdmin, admin store.User, msg, errMsg, open string,
|
||||
_ AccountAdmin, admin store.User, msg, errMsg, open string,
|
||||
) {
|
||||
p.renderPeople(w, r, admin, peopleView{Tab: tabPeople, Msg: msg, Err: errMsg, Open: open})
|
||||
}
|
||||
|
||||
// buildUsers is the tree and its people, as the first tab shows them.
|
||||
func (p *AccountPages) buildUsers(r *http.Request, s AccountAdmin, admin store.User, msg, errMsg, open string) adminUsersData {
|
||||
ctx := r.Context()
|
||||
d := adminUsersData{
|
||||
Nav: navFrom(r, "people"),
|
||||
Message: msg, Error: errMsg, Open: open, AdminID: admin.ID,
|
||||
Message: msg, Error: errMsg, Open: open, AdminID: admin.ID, Sel: r.FormValue("sel"),
|
||||
CSRF: csrfFor(r),
|
||||
Version: p.version,
|
||||
HasMirror: p.hasMirror,
|
||||
@@ -558,7 +567,7 @@ func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
|
||||
d.Scopes = append(d.Scopes, scopeOption{"project:" + pp.ID, pp.Path})
|
||||
}
|
||||
sort.Slice(d.Scopes, func(i, j int) bool { return d.Scopes[i].Label < d.Scopes[j].Label })
|
||||
render(w, "admin_users", d)
|
||||
return d
|
||||
}
|
||||
|
||||
func keyState(k store.Key, now time.Time) string {
|
||||
|
||||
@@ -18,6 +18,33 @@ type ChatModel struct {
|
||||
// offered anyway: the first message is what asks, so hiding it until it
|
||||
// runs meant it could never be started from here.
|
||||
OnDemand bool
|
||||
// Hardware is what the model's running replicas compute on: "cpu", or a
|
||||
// GPU class. Several when it runs on both, and empty when nothing runs
|
||||
// or nothing said.
|
||||
Hardware []string
|
||||
}
|
||||
|
||||
// Runs names the hardware for the picker: "GPU", "CPU", or "GPU + CPU" for a
|
||||
// model with replicas on both, where which one answers is the router's choice
|
||||
// and each reply says which it was.
|
||||
func (m ChatModel) Runs() string {
|
||||
var gpu, cpu bool
|
||||
for _, h := range m.Hardware {
|
||||
if h == "cpu" {
|
||||
cpu = true
|
||||
} else {
|
||||
gpu = true
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case gpu && cpu:
|
||||
return "GPU + CPU"
|
||||
case gpu:
|
||||
return "GPU"
|
||||
case cpu:
|
||||
return "CPU"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ChatContext is one thing the chat page can spend against: a project, or the
|
||||
|
||||
@@ -185,3 +185,19 @@ func TestTheTruncationNoticeIsNotPutInsideTheReply(t *testing.T) {
|
||||
t.Errorf("the reply is written as HTML other than from /chat/render (%d assignments)", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChatModelRunsNamesTheHardware(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
hw []string
|
||||
want string
|
||||
}{
|
||||
{nil, ""},
|
||||
{[]string{"cpu"}, "CPU"},
|
||||
{[]string{"6000-24g"}, "GPU"},
|
||||
{[]string{"cpu", "6000-24g"}, "GPU + CPU"},
|
||||
} {
|
||||
if got := (ChatModel{Hardware: c.hw}).Runs(); got != c.want {
|
||||
t.Errorf("%v: %q, want %q", c.hw, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,13 @@ type DashboardData struct {
|
||||
CPU cpuCapacity
|
||||
// Menu is the shared top menu; Nav below is what the cluster page adds.
|
||||
Menu Nav
|
||||
// GPUMap is every card coloured by one figure, and IdleGPUs the replicas
|
||||
// holding cards that are doing nothing; both from the last heartbeats.
|
||||
GPUMap *gpuMap
|
||||
IdleGPUs []idleHolding
|
||||
// Efficiency is what each model gets from its cards, from the metrics
|
||||
// store; nil without one.
|
||||
Efficiency []modelEfficiency
|
||||
// Notice and Problem are what the last change made from this page did,
|
||||
// carried over the redirect back to it.
|
||||
Notice, Problem string
|
||||
@@ -198,11 +205,16 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
d.AnyFabrics = d.AnyFabrics || len(n.Fabrics) > 0
|
||||
}
|
||||
d.NodePrefix = sharedPrefix(ids)
|
||||
if d.AnyGPUs {
|
||||
d.GPUMap = buildGPUMap(d.Nodes, d.Replicas, r.URL.Query().Get("gpumap"), d.NodePrefix)
|
||||
d.IdleGPUs = idleHoldings(d.Nodes, d.Replicas, d.NodePrefix)
|
||||
}
|
||||
for _, m := range d.Models {
|
||||
d.AnyFallbacks = d.AnyFallbacks || len(m.Fallbacks) > 0
|
||||
}
|
||||
if d.AnyGPUs {
|
||||
d.Fleet.LoadGPU(r.Context(), nav.History, nav.Window)
|
||||
d.Efficiency = loadEfficiency(r.Context(), nav.History)
|
||||
}
|
||||
d.Summary = src.Summary()
|
||||
if fs, ok := src.(FabricSource); ok {
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// modelEfficiency is what a model gets out of the GPUs it holds, over the
|
||||
// last quarter hour.
|
||||
type modelEfficiency struct {
|
||||
Model string
|
||||
// TokensPerSec is output tokens a second; GPUs the cards it holds now.
|
||||
TokensPerSec float64
|
||||
GPUs float64
|
||||
// Watts is what those cards draw; Util their average utilisation.
|
||||
Watts, Util float64
|
||||
// PerGPU and PerWh are the two ways of saying how much work the cards
|
||||
// do: output tokens a second per card, and per watt-hour drawn.
|
||||
PerGPU, PerWh float64
|
||||
}
|
||||
|
||||
// The queries behind the table. Each model's cards are found by joining the
|
||||
// GPU series to llmgw_gpu_assigned on the card, which carries the model.
|
||||
var efficiencyQueries = []struct {
|
||||
expr string
|
||||
into func(*modelEfficiency, float64)
|
||||
}{
|
||||
{`sum by (model) (rate(llmgw_tokens_total{direction="completion"}[15m]))`,
|
||||
func(e *modelEfficiency, v float64) { e.TokensPerSec = v }},
|
||||
{`count by (model) (llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.GPUs = v }},
|
||||
{`sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.Watts = v }},
|
||||
{`avg by (model) (llmgw_gpu_utilization_percent * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
|
||||
func(e *modelEfficiency, v float64) { e.Util = v }},
|
||||
}
|
||||
|
||||
// loadEfficiency is the table, best per card first; nil when the metrics
|
||||
// store cannot be read or no model holds a GPU.
|
||||
func loadEfficiency(ctx context.Context, h History) []modelEfficiency {
|
||||
if h == nil {
|
||||
return nil
|
||||
}
|
||||
byModel := map[string]*modelEfficiency{}
|
||||
for _, q := range efficiencyQueries {
|
||||
// The last of a short range: what History offers, and the one point
|
||||
// the table needs.
|
||||
series, err := h.RangeQuery(ctx, q.expr, "model", 10*time.Minute, 5*time.Minute)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
for _, s := range series {
|
||||
if len(s.Points) == 0 {
|
||||
continue
|
||||
}
|
||||
e := byModel[s.Name]
|
||||
if e == nil {
|
||||
e = &modelEfficiency{Model: s.Name}
|
||||
byModel[s.Name] = e
|
||||
}
|
||||
q.into(e, s.Last())
|
||||
}
|
||||
}
|
||||
var out []modelEfficiency
|
||||
for _, e := range byModel {
|
||||
if e.GPUs == 0 {
|
||||
continue // a model on the CPU; this table is about cards
|
||||
}
|
||||
e.PerGPU = e.TokensPerSec / e.GPUs
|
||||
if e.Watts > 0 {
|
||||
e.PerWh = e.TokensPerSec * 3600 / e.Watts
|
||||
}
|
||||
out = append(out, *e)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].PerGPU != out[j].PerGPU {
|
||||
return out[i].PerGPU > out[j].PerGPU
|
||||
}
|
||||
return out[i].Model < out[j].Model
|
||||
})
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"html/template"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// The GPU map is every card in the fleet, a square each, a row per node,
|
||||
// coloured by one figure: how busy, how hot, or how full its memory is. A
|
||||
// hundred cards in a table are a hundred rows to read; on the map the one
|
||||
// that is not like the others is seen before anything is read.
|
||||
|
||||
// gpuMapModes are what the map can be coloured by.
|
||||
var gpuMapModes = []struct{ Key, Label string }{
|
||||
{"util", "busy"}, {"memory", "memory"}, {"temp", "temperature"},
|
||||
}
|
||||
|
||||
type gpuMap struct {
|
||||
By string
|
||||
Modes []struct{ Key, Label string }
|
||||
Rows []gpuMapRow
|
||||
// Legend says what the colours mean in this mode.
|
||||
Legend string
|
||||
}
|
||||
|
||||
type gpuMapRow struct {
|
||||
Node string
|
||||
Cells []gpuCell
|
||||
}
|
||||
|
||||
type gpuCell struct {
|
||||
Index int
|
||||
// Model and Replica hold the card; empty is a free card.
|
||||
Model, Replica string
|
||||
// Style is the cell's colour, built here from numbers alone, which is
|
||||
// what makes it safe to put in a style attribute; Title says everything
|
||||
// known about the card.
|
||||
Style template.CSS
|
||||
Title string
|
||||
// Label is the figure written in the cell.
|
||||
Label string
|
||||
// Faulty is a card whose wear needs seeing to.
|
||||
Faulty bool
|
||||
}
|
||||
|
||||
// idleHolding is a replica holding GPUs that are doing nothing.
|
||||
type idleHolding struct {
|
||||
Model, Replica, Node string
|
||||
GPUs []int
|
||||
// MaxUtil is the busiest of its cards, now.
|
||||
MaxUtil float64
|
||||
// Idle is how long since a request was routed to it, or "" when not
|
||||
// known.
|
||||
Idle string
|
||||
}
|
||||
|
||||
// buildGPUMap draws the map from what the nodes last reported and where
|
||||
// replicas are. Nil for a fleet without GPUs.
|
||||
func buildGPUMap(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, by, prefix string) *gpuMap {
|
||||
held := heldBy(reps)
|
||||
m := &gpuMap{By: by, Modes: gpuMapModes}
|
||||
if m.By == "" {
|
||||
m.By = "util"
|
||||
}
|
||||
switch m.By {
|
||||
case "temp":
|
||||
m.Legend = "Blue is cool, red is 90 °C and over."
|
||||
case "memory":
|
||||
m.Legend = "Pale is empty, dark is full. Engines reserve most of a card's memory when they start."
|
||||
default:
|
||||
m.By = "util"
|
||||
m.Legend = "Pale is idle, dark green is fully busy. A dashed border is a card no replica holds."
|
||||
}
|
||||
for _, n := range nodes {
|
||||
if n.GPUTotal == 0 {
|
||||
continue
|
||||
}
|
||||
row := gpuMapRow{Node: strings.TrimPrefix(n.ID, prefix)}
|
||||
for i := range n.GPUTotal {
|
||||
c := gpuCell{Index: i}
|
||||
if h, ok := held[gpuKey(n.ID, i)]; ok {
|
||||
c.Model, c.Replica = h.model, h.replica
|
||||
}
|
||||
var title []string
|
||||
title = append(title, fmt.Sprintf("%s GPU %d", n.ID, i))
|
||||
if c.Model != "" {
|
||||
title = append(title, c.Model+" ("+c.Replica+")")
|
||||
} else {
|
||||
title = append(title, "free")
|
||||
}
|
||||
s, sampled := sensorOf(n, i)
|
||||
if sampled {
|
||||
if s.MemoryTotalMB > 0 && s.UtilPercent >= 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f%% busy", s.UtilPercent))
|
||||
}
|
||||
if s.MemoryTotalMB > 0 {
|
||||
title = append(title, fmt.Sprintf("%.1f of %.0f GB", s.MemoryUsedMB/1024, s.MemoryTotalMB/1024))
|
||||
}
|
||||
if s.TempC > 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f °C", s.TempC))
|
||||
}
|
||||
if s.PowerW > 0 {
|
||||
title = append(title, fmt.Sprintf("%.0f W", s.PowerW))
|
||||
}
|
||||
if len(s.Throttle) > 0 {
|
||||
title = append(title, "held back by "+strings.Join(s.Throttle, ", "))
|
||||
}
|
||||
title = append(title, s.Problems...)
|
||||
}
|
||||
c.Title = strings.Join(title, " · ")
|
||||
style, label := cellColour(m.By, s, sampled)
|
||||
if c.Model == "" {
|
||||
style += ";border-style:dashed"
|
||||
}
|
||||
// A card wearing out is outlined whatever the map is coloured
|
||||
// by: it is the one thing on the map that needs doing.
|
||||
if len(s.Problems) > 0 {
|
||||
style += ";border:2px solid #c0392b"
|
||||
c.Faulty = true
|
||||
}
|
||||
c.Style, c.Label = template.CSS(style), label
|
||||
row.Cells = append(row.Cells, c)
|
||||
}
|
||||
m.Rows = append(m.Rows, row)
|
||||
}
|
||||
if len(m.Rows) == 0 {
|
||||
return nil
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// cellColour is a cell's background, and the figure written in it.
|
||||
func cellColour(by string, s sensorView, sampled bool) (string, string) {
|
||||
const unknown = "background:var(--panel-2,#eee)"
|
||||
switch by {
|
||||
case "temp":
|
||||
if !sampled || s.TempC <= 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
// 30 °C blue to 90 °C red.
|
||||
f := clamp01((s.TempC - 30) / 60)
|
||||
return fmt.Sprintf("background:hsl(%.0f,70%%,55%%)", 220-220*f), fmt.Sprintf("%.0f", s.TempC)
|
||||
case "memory":
|
||||
if !sampled || s.MemoryTotalMB <= 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
f := clamp01(s.MemoryUsedMB / s.MemoryTotalMB)
|
||||
return fmt.Sprintf("background:hsl(265,45%%,%.0f%%)", 90-55*f), fmt.Sprintf("%.0f", 100*f)
|
||||
}
|
||||
if !sampled || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
|
||||
return unknown, ""
|
||||
}
|
||||
f := clamp01(s.UtilPercent / 100)
|
||||
return fmt.Sprintf("background:hsl(140,%.0f%%,%.0f%%)", 15+50*f, 88-50*f), fmt.Sprintf("%.0f", s.UtilPercent)
|
||||
}
|
||||
|
||||
// idleHoldings are the replicas whose GPUs all read under 5% busy now:
|
||||
// capacity held and not asked for, the first place to find room.
|
||||
func idleHoldings(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, prefix string) []idleHolding {
|
||||
byID := map[string]adminapi.NodeView{}
|
||||
for _, n := range nodes {
|
||||
byID[n.ID] = n
|
||||
}
|
||||
var out []idleHolding
|
||||
for _, r := range reps {
|
||||
if r.State != "ready" {
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
if len(sh.GPUs) == 0 {
|
||||
continue
|
||||
}
|
||||
h := idleHolding{Model: r.Model, Replica: r.ID, Node: strings.TrimPrefix(sh.Node, prefix), GPUs: sh.GPUs}
|
||||
idle, known := true, true
|
||||
for _, g := range sh.GPUs {
|
||||
s, ok := sensorOf(byID[sh.Node], g)
|
||||
if !ok || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
|
||||
known = false
|
||||
break
|
||||
}
|
||||
h.MaxUtil = max(h.MaxUtil, s.UtilPercent)
|
||||
if s.UtilPercent >= 5 {
|
||||
idle = false
|
||||
}
|
||||
}
|
||||
if !known || !idle {
|
||||
continue
|
||||
}
|
||||
if r.IdleSec >= 0 {
|
||||
h.Idle = (time.Duration(r.IdleSec) * time.Second).String()
|
||||
}
|
||||
out = append(out, h)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool { return len(out[i].GPUs) > len(out[j].GPUs) })
|
||||
return out
|
||||
}
|
||||
|
||||
type holder struct{ model, replica string }
|
||||
|
||||
func heldBy(reps []adminapi.ReplicaView) map[string]holder {
|
||||
out := map[string]holder{}
|
||||
for _, r := range reps {
|
||||
switch r.State {
|
||||
case "stopped", "failed":
|
||||
continue
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
for _, g := range sh.GPUs {
|
||||
out[gpuKey(sh.Node, g)] = holder{r.Model, r.ID}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func gpuKey(node string, gpu int) string { return fmt.Sprintf("%s/%d", node, gpu) }
|
||||
|
||||
// sensorView is a card's last sample.
|
||||
type sensorView = struct {
|
||||
TempC, PowerW, UtilPercent, MemoryUsedMB, MemoryTotalMB float64
|
||||
Throttle []string
|
||||
Problems []string
|
||||
}
|
||||
|
||||
func sensorOf(n adminapi.NodeView, gpu int) (sensorView, bool) {
|
||||
if n.Load == nil {
|
||||
return sensorView{}, false
|
||||
}
|
||||
for _, s := range n.Load.GPUs {
|
||||
if s.Index == gpu {
|
||||
return sensorView{TempC: s.TempC, PowerW: s.PowerW, UtilPercent: s.UtilPercent,
|
||||
MemoryUsedMB: s.MemoryUsedMB, MemoryTotalMB: s.MemoryTotalMB, Throttle: s.Throttle,
|
||||
Problems: s.Wear.Problems()}, true
|
||||
}
|
||||
}
|
||||
return sensorView{}, false
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
)
|
||||
|
||||
// mapFleet is a node of four cards: chat holds two and is idle, code holds
|
||||
// one and is busy, one is free.
|
||||
type mapFleet struct{ fakeSource }
|
||||
|
||||
func (mapFleet) Nodes() []adminapi.NodeView {
|
||||
busy := func(i int, util, temp float64) inventory.GPUSensor {
|
||||
return inventory.GPUSensor{Index: i, UtilPercent: util, TempC: temp, MemoryUsedMB: 40000, MemoryTotalMB: 81559, PowerW: 300}
|
||||
}
|
||||
return []adminapi.NodeView{{ID: "gpu07", State: "ready", GPUTotal: 4, GPUFree: 1, Classes: []string{"h100-80g"},
|
||||
Load: &adminapi.LoadView{GPUs: []inventory.GPUSensor{busy(0, 1, 35), busy(1, 0, 34), busy(2, 93, 81), worn(busy(3, 0, 30))}}}}
|
||||
}
|
||||
|
||||
func worn(s inventory.GPUSensor) inventory.GPUSensor {
|
||||
s.Wear = &inventory.GPUWear{RemapPending: true, XIDs: 1, LastXID: 79, LastXIDText: "GPU has fallen off the bus."}
|
||||
return s
|
||||
}
|
||||
|
||||
func (mapFleet) Replicas(string) []adminapi.ReplicaView {
|
||||
return []adminapi.ReplicaView{
|
||||
{ID: "chat-1", Model: "chat", State: "ready", IdleSec: 1800, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{0, 1}}}},
|
||||
{ID: "code-1", Model: "code", State: "ready", IdleSec: 2, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{2}}}},
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClusterPageMapsEveryCardAndNamesTheIdleOnes(t *testing.T) {
|
||||
_, body := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if n := strings.Count(body, `class="gpu-cell"`); n != 4 {
|
||||
t.Errorf("%d cards on the map, want 4", n)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"gpu07 GPU 2 · code (code-1) · 93% busy",
|
||||
"gpu07 GPU 3 · free",
|
||||
"memory rows remapped, waiting for a reset · XID 79 (GPU has fallen off the bus.)",
|
||||
"border:2px solid #c0392b",
|
||||
"border-style:dashed",
|
||||
"Cards held and doing nothing",
|
||||
"<code>chat-1</code></td><td><code>gpu07</code>", "30m0s ago",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "<code>code-1</code></td><td><code>gpu07</code>") {
|
||||
t.Error("a busy replica was listed as idle")
|
||||
}
|
||||
_, temp := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/?gpumap=temp", nil)
|
||||
if !strings.Contains(temp, "<strong>temperature</strong>") || !strings.Contains(temp, ">81</span>") {
|
||||
t.Error("the map coloured by temperature does not say so, or show it")
|
||||
}
|
||||
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(cpu, "gpu-map") {
|
||||
t.Error("a fleet without GPUs has a GPU map")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachModelSaysWhatItGetsFromItsCards(t *testing.T) {
|
||||
h := &fakeHistory{byExpr: map[string][]promq.Series{}}
|
||||
for i, vals := range [][2]float64{{120, 30}, {2, 2}, {1200, 400}, {80, 40}} {
|
||||
h.byExpr[efficiencyQueries[i].expr] = []promq.Series{ts("chat", vals[0], vals[0]), ts("code", vals[1], vals[1])}
|
||||
}
|
||||
got := loadEfficiency(context.Background(), h)
|
||||
if len(got) != 2 || got[0].Model != "chat" {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
c := got[0]
|
||||
if c.PerGPU != 60 || c.PerWh != 120*3600/1200.0 || c.Util != 80 {
|
||||
t.Errorf("chat: %+v", c)
|
||||
}
|
||||
if got[1].PerGPU != 15 {
|
||||
t.Errorf("code: %+v", got[1])
|
||||
}
|
||||
}
|
||||
@@ -25,7 +25,8 @@ func TestEveryPageDrawsTheSameMenu(t *testing.T) {
|
||||
p.RegisterAdminMirror(mux, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{}}})
|
||||
|
||||
pages := map[string]string{
|
||||
"/admin/users": "People", "/admin/quotas": "Quotas", "/admin/mirror": "Mirror",
|
||||
// Limits and usage are tabs of the people page, which they mark.
|
||||
"/admin/users": "People", "/admin/quotas": "People", "/admin/mirror": "Mirror",
|
||||
}
|
||||
var first string
|
||||
for path, current := range pages {
|
||||
@@ -44,7 +45,7 @@ func TestEveryPageDrawsTheSameMenu(t *testing.T) {
|
||||
t.Errorf("%s draws a different menu:\n%s\nfrom:\n%s", path, plain, first)
|
||||
}
|
||||
}
|
||||
for _, link := range []string{"/admin", "/admin/users", "/admin/quotas", "/admin/usage",
|
||||
for _, link := range []string{"/admin", "/admin/users",
|
||||
"/admin/mirror", "/chat", "/profile", "/logout"} {
|
||||
if !strings.Contains(first, `"`+link+`"`) {
|
||||
t.Errorf("the menu lacks %s", link)
|
||||
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
@@ -70,6 +71,9 @@ type orgPageData struct {
|
||||
func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced Priced) {
|
||||
p.orgs = s
|
||||
mux.HandleFunc("GET /org", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
|
||||
if p.toPeople(w, r, user, "") {
|
||||
return
|
||||
}
|
||||
orgs, err := p.viewableOrgs(r.Context(), s, user)
|
||||
if err == nil && len(orgs) == 0 && user.IsAdmin {
|
||||
// Said, not a 404: an administrator following the link on a
|
||||
@@ -89,6 +93,9 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
|
||||
}))
|
||||
mux.HandleFunc("GET /org/{id}", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
|
||||
id := r.PathValue("id")
|
||||
if p.toPeople(w, r, user, id) {
|
||||
return
|
||||
}
|
||||
if !user.IsAdmin {
|
||||
ok, err := s.ManagesOrg(r.Context(), user.ID, id)
|
||||
if err != nil || !ok {
|
||||
@@ -102,6 +109,22 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
|
||||
}))
|
||||
}
|
||||
|
||||
// toPeople sends an administrator to the people page, which shows an
|
||||
// organization's members, limits and use beside the whole tree and can change
|
||||
// them; this page stays for the managers who may not see that one. It says
|
||||
// whether it did.
|
||||
func (p *AccountPages) toPeople(w http.ResponseWriter, r *http.Request, user store.User, orgID string) bool {
|
||||
if !user.IsAdmin || p.peopleStore == nil {
|
||||
return false
|
||||
}
|
||||
to := "/admin/users"
|
||||
if orgID != "" {
|
||||
to += "?sel=org:" + url.QueryEscape(orgID)
|
||||
}
|
||||
http.Redirect(w, r, to, http.StatusSeeOther)
|
||||
return true
|
||||
}
|
||||
|
||||
// viewableOrgs is every organization an administrator may look at, or the
|
||||
// ones a person manages.
|
||||
func (p *AccountPages) viewableOrgs(ctx context.Context, s OrgViewer, user store.User) ([]store.Org, error) {
|
||||
|
||||
@@ -124,3 +124,22 @@ func TestWithNoOrganizationsAnAdministratorIsToldSo(t *testing.T) {
|
||||
t.Error("the menu links to organizations when there are none")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdministratorIsSentToThePeoplePageForAnOrganization(t *testing.T) {
|
||||
// Its members, limits and use are a panel there, beside the whole tree
|
||||
// and changeable; /org stays for the managers who cannot see that page.
|
||||
p := NewAccountPages(fakeAccounts{user: store.User{ID: "a", IsAdmin: true}}, nil, nil, "v0",
|
||||
slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsers(mux, &fakeAdmin{})
|
||||
p.RegisterOrgPages(mux, fakeOrgs{}, nil)
|
||||
for path, to := range map[string]string{"/org": "/admin/users", "/org/org-s": "/admin/users?sel=org:org-s"} {
|
||||
w := mirrorGet(t, mux, path)
|
||||
if w.Code != http.StatusSeeOther || w.Header().Get("Location") != to {
|
||||
t.Errorf("%s: %d to %q, want %s", path, w.Code, w.Header().Get("Location"), to)
|
||||
}
|
||||
}
|
||||
if menu := menuRe.FindString(mirrorGet(t, mux, "/admin/users").Body.String()); strings.Contains(menu, `href="/org"`) {
|
||||
t.Error("the administrator's menu still offers /org beside People")
|
||||
}
|
||||
}
|
||||
@@ -862,3 +862,43 @@ table.rows td.warn { color: var(--warn); cursor: help; }
|
||||
.held-card .adapters h4 { font-size: .9rem; margin: 0 0 .35rem; display: flex; gap: .75rem; align-items: baseline; }
|
||||
.held-card .adapter-form .field-row { align-items: flex-end; flex-wrap: wrap; gap: .5rem 1rem; }
|
||||
.held-card .adapter-form select { display: block; margin-top: .25rem; max-width: 28rem; }
|
||||
|
||||
/* The GPU map: a square per card, a row per node. */
|
||||
.gpu-map { display: flex; flex-direction: column; gap: 3px; margin: 6px 0; overflow-x: auto; }
|
||||
.gpu-row { display: flex; align-items: center; gap: 3px; }
|
||||
.gpu-node { width: 7em; flex: none; font-size: 12px; color: var(--muted, #666); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.gpu-cell { width: 26px; height: 26px; flex: none; border: 1px solid rgba(0,0,0,.18); border-radius: 4px;
|
||||
font-size: 10px; line-height: 26px; text-align: center; color: rgba(0,0,0,.7); cursor: default; }
|
||||
|
||||
/* The people page: one page in tabs, and a panel for the branch or person
|
||||
chosen in the tree, beside it on a wide screen and above it on a narrow one. */
|
||||
.page-tabs { display: flex; flex-wrap: wrap; gap: .25rem; border-bottom: 1px solid var(--line); margin: .25rem 0 1rem; }
|
||||
.page-tabs a { padding: .45rem .8rem; color: var(--muted); text-decoration: none; border-bottom: 2px solid transparent; margin-bottom: -1px; }
|
||||
.page-tabs a:hover { color: var(--fg); }
|
||||
.page-tabs a[aria-current="page"] { color: var(--fg); border-bottom-color: var(--accent); font-weight: 600; }
|
||||
.people-grid.with-detail { display: grid; gap: 1.25rem; grid-template-columns: minmax(0, 1fr) minmax(0, 24rem); align-items: start; }
|
||||
.people-grid.with-detail .people-detail { grid-column: 2; grid-row: 1; position: sticky; top: .75rem; max-height: calc(100vh - 1.5rem); overflow-y: auto; }
|
||||
.people-grid.with-detail .people-main { grid-column: 1; grid-row: 1; min-width: 0; }
|
||||
@media (max-width: 960px) {
|
||||
.people-grid.with-detail { grid-template-columns: minmax(0, 1fr); }
|
||||
.people-grid.with-detail .people-detail { grid-column: 1; grid-row: 1; position: static; max-height: none; }
|
||||
.people-grid.with-detail .people-main { grid-row: 2; }
|
||||
}
|
||||
.people-detail { background: var(--card); border: 1px solid var(--line); border-radius: 8px; padding: .9rem 1rem; }
|
||||
.people-detail h2 { margin: .1rem 0 0; font-size: 1.15rem; }
|
||||
.people-detail h3 { margin: 1rem 0 .4rem; font-size: .9rem; text-transform: uppercase; letter-spacing: .04em; color: var(--muted); }
|
||||
.people-detail .chart-wrap { margin-bottom: .75rem; }
|
||||
.people-detail .q-form .field-row { flex-wrap: wrap; }
|
||||
.pd-head { position: relative; padding-right: 1.5rem; }
|
||||
.pd-path, .pd-sub { margin: .2rem 0 0; font-size: .85rem; color: var(--muted); }
|
||||
.pd-close { position: absolute; top: 0; right: 0; font-size: 1.3rem; line-height: 1; color: var(--muted); text-decoration: none; }
|
||||
.pd-limits { display: grid; grid-template-columns: auto 1fr; gap: .25rem .9rem; margin: 0; font-size: .88rem; }
|
||||
.pd-limits dt { color: var(--muted); }
|
||||
.pd-limits dd { margin: 0; font-variant-numeric: tabular-nums; }
|
||||
a.sel-link { color: inherit; text-decoration: none; border-bottom: 1px dotted var(--line); }
|
||||
a.sel-link:hover { color: var(--accent); border-bottom-color: var(--accent); }
|
||||
a.sel-link.on { color: var(--accent); border-bottom: 2px solid var(--accent); }
|
||||
a.sel-link.small { font-size: .78rem; color: var(--muted); margin-left: .4rem; }
|
||||
.people-detail h3.chart-title { text-transform: none; letter-spacing: 0; color: var(--fg); font-size: .85rem; margin: .2rem 0; }
|
||||
.people-detail details.edit { display: inline-block; margin-right: 1rem; vertical-align: top; }
|
||||
.people-detail details.edit[open] { display: block; margin: .5rem 0; }
|
||||
@@ -0,0 +1,148 @@
|
||||
{{/* People, their limits and their use: one page in three tabs, because all
|
||||
three are about the same tree. Each tab keeps its old address. Forms and
|
||||
links work as plain navigation; htmx, when it loads, swaps only #people,
|
||||
so a change does not jump the page. */}}
|
||||
{{define "admin_people"}}<!doctype html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>People · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
||||
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false}'>
|
||||
<script src="/_static/htmx-2.0.11.min.js" defer></script>
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>People</h1>
|
||||
<p class="sub">Organizations, teams, projects and the people in them: what each may use, and what it used</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
|
||||
<div id="people" hx-boost="true" hx-target="#people" hx-select="#people"
|
||||
hx-swap="outerHTML" hx-push-url="false">
|
||||
|
||||
<nav class="page-tabs" aria-label="People">
|
||||
{{if .HasPeople}}<a href="/admin/users" hx-push-url="true"{{if eq .Tab "people"}} aria-current="page"{{end}}>Organizations & people{{if .Pending}} <span class="tag">{{.Pending}} waiting</span>{{end}}</a>{{end}}
|
||||
{{if .HasLimits}}<a href="/admin/quotas" hx-push-url="true"{{if eq .Tab "limits"}} aria-current="page"{{end}}>Limits</a>{{end}}
|
||||
{{if .HasUsage}}<a href="/admin/usage" hx-push-url="true"{{if eq .Tab "usage"}} aria-current="page"{{end}}>Usage</a>{{end}}
|
||||
</nav>
|
||||
|
||||
{{if .Users}}
|
||||
<div class="people-grid{{if .Detail}} with-detail{{end}}">
|
||||
{{with .Detail}}<aside class="people-detail" id="detail" aria-label="{{.Name}}">{{template "people-detail" .}}</aside>{{end}}
|
||||
<div class="people-main">{{template "people_body" .Users}}</div>
|
||||
</div>
|
||||
{{else if .Quotas}}{{template "limits_body" .Quotas}}
|
||||
{{else if .Usage}}{{template "usage_body" .Usage}}
|
||||
{{end}}
|
||||
</div>
|
||||
|
||||
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
|
||||
</div>
|
||||
<script>
|
||||
// Search the account list. Outside #people so what was typed survives a swap,
|
||||
// and applied again after every swap.
|
||||
(function () {
|
||||
var box = document.createElement('input');
|
||||
box.type = 'search'; box.placeholder = 'Find a person by email, name or project';
|
||||
box.className = 'search'; box.setAttribute('aria-label', 'Find a person');
|
||||
function apply() {
|
||||
var q = box.value.trim().toLowerCase();
|
||||
document.querySelectorAll('#account-list tr[data-q]').forEach(function (tr) {
|
||||
tr.hidden = q !== '' && tr.dataset.q.indexOf(q) < 0;
|
||||
});
|
||||
}
|
||||
box.addEventListener('input', apply);
|
||||
function place() {
|
||||
var h = document.getElementById('accounts');
|
||||
if (h && box.parentNode !== h.parentNode) h.after(box);
|
||||
apply();
|
||||
}
|
||||
place();
|
||||
// Projects the reader had open stay open across a change.
|
||||
var open = [];
|
||||
document.addEventListener('htmx:beforeSwap', function () {
|
||||
open = Array.prototype.map.call(document.querySelectorAll('details.project[open]'), function (d) { return d.id; });
|
||||
});
|
||||
document.addEventListener('htmx:afterSettle', function () {
|
||||
open.forEach(function (id) { var d = document.getElementById(id); if (d) d.open = true; });
|
||||
place();
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body></html>{{end}}
|
||||
|
||||
{{/* One branch or person: its limits, its capacity and its use, together. */}}
|
||||
{{define "people-detail"}}{{$d := .}}{{$sel := printf "%s:%s" .Kind .ID}}
|
||||
<div class="pd-head">
|
||||
<span class="q-kind">{{if eq (printf "%s" .Kind) "user"}}person{{else}}{{.Kind}}{{end}}</span>
|
||||
<h2>{{.Name}}</h2>
|
||||
{{if and .Path (ne .Path .Name)}}<p class="pd-path">{{.Path}}</p>{{end}}
|
||||
<a href="/admin/users" class="pd-close" aria-label="Close">×</a>
|
||||
</div>
|
||||
|
||||
{{with .Memberships}}<p class="pd-sub">In {{range $i, $m := .}}{{if $i}}, {{end}}<a href="/admin/users?sel={{$m.Scope}}:{{$m.ScopeID}}">{{$m.Path}}</a>{{end}}</p>
|
||||
{{else}}{{if eq (printf "%s" $d.Kind) "user"}}<p class="pd-sub">On their own: in no organization, team or project.</p>{{end}}{{end}}
|
||||
|
||||
{{if or .Limits .Person}}
|
||||
<h3>Limits</h3>
|
||||
{{if not .BudgetsBind}}<p class="note">No model has a price, so budgets stay at zero; rate and token limits apply.</p>{{end}}
|
||||
{{with .Limits}}
|
||||
<dl class="pd-limits">
|
||||
<dt>Requests/min</dt><dd>{{template "quota-value" dict "Own" .RPM "Up" .RPMUp "From" .RPMFrom}}</dd>
|
||||
<dt>Tokens/min</dt><dd>{{template "quota-value" dict "Own" .TPM "Up" .TPMUp "From" .TPMFrom}}</dd>
|
||||
<dt>Budget</dt><dd>{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}{{else}}{{template "quota-value" dict "Own" "" "Up" .BudgetUp "From" .BudgetFrom}}{{end}}</dd>
|
||||
<dt>Spent, {{$d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{.Percent}}%</span>{{end}}</dd>
|
||||
<dt>Models</dt><dd>{{if .Models}}only <code>{{.Models}}</code>{{else if .ModelsFrom}}limited by {{.ModelsFrom}}{{else}}every model{{end}}</dd>
|
||||
{{if .Member}}<dt>Each member</dt><dd>{{.Member}}</dd>{{end}}
|
||||
</dl>
|
||||
{{if .Shadowed}}<p class="q-sub q-warn">{{.Shadowed}}</p>{{end}}
|
||||
<details class="edit"><summary>{{if .HasAny}}change{{else}}set{{end}} limits</summary>
|
||||
{{template "quota-form" (dict "Scope" (printf "%s" .Scope) "ID" .ID "CSRF" $d.CSRF "Sel" $sel
|
||||
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
|
||||
"Title" (printf "Quota for %s" .Crumb)
|
||||
"Note" "Shared by everything beneath it.")}}
|
||||
</details>
|
||||
<details class="edit"><summary>limit each member</summary>
|
||||
{{template "quota-form" (dict "Scope" (printf "%s-member" .Scope) "ID" .ID "F" .MemberRaw "CSRF" $d.CSRF "Sel" $sel
|
||||
"Title" (printf "Each member of %s, separately" .Crumb)
|
||||
"Note" "Every member gets this much here, counted apart from the others.")}}
|
||||
</details>
|
||||
{{end}}
|
||||
{{with .Person}}
|
||||
<dl class="pd-limits">
|
||||
<dt>Requests/min</dt><dd>{{or .RPM "—"}}</dd>
|
||||
<dt>Tokens/min</dt><dd>{{or .TPM "—"}}</dd>
|
||||
<dt>Budget</dt><dd>{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}{{else}}—{{end}}</dd>
|
||||
<dt>Spent, {{$d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{.Percent}}%</span>{{end}}</dd>
|
||||
<dt>Models</dt><dd>{{if .Models}}only <code>{{.Models}}</code>{{else}}every model{{end}}</dd>
|
||||
</dl>
|
||||
<details class="edit"><summary>{{if .HasAny}}change{{else}}set{{end}} limits</summary>
|
||||
{{template "quota-form" (dict "Scope" "user" "ID" .ID "CSRF" $d.CSRF "Sel" $sel
|
||||
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
|
||||
"Title" (printf "%s, everywhere" .Email)
|
||||
"Note" "Across every key this person holds, every project they use and the chat page.")}}
|
||||
</details>
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if .HasCapacity}}
|
||||
<h3>Capacity</h3>
|
||||
{{with .Capacity}}
|
||||
<dl class="pd-limits">
|
||||
<dt>GPUs</dt><dd>{{or .GPUs "—"}}</dd>
|
||||
<dt>Cores</dt><dd>{{or .Cores "—"}}</dd>
|
||||
<dt>Memory GB</dt><dd>{{or .Memory "—"}}</dd>
|
||||
</dl>
|
||||
<p class="note">Held of guaranteed (at most the limit). <a href="/admin/quotas?cap={{$sel}}#capacity">change</a></p>
|
||||
{{else}}<p class="note">No capacity quota: its models share the cluster by their class alone.
|
||||
<a href="/admin/quotas?cap={{$sel}}#capacity">set one</a></p>{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if .HasUsage}}
|
||||
<h3>Use, {{.UsageWindow}}</h3>
|
||||
{{if .Totals.Name}}
|
||||
<p class="pd-sub">{{compact .Totals.Requests}} requests · {{compact .Totals.Tokens}} tokens{{if .Priced}} · {{.Totals.Spend}} spent{{end}}{{if .Totals.Errors}} · {{.Totals.Errors}} errors{{end}}</p>
|
||||
{{template "tschart" dict "t" .Requests "title" "Requests" "note" ""}}
|
||||
{{template "tschart" dict "t" .Tokens "title" "Tokens" "note" ""}}
|
||||
{{if .Priced}}{{template "tschart" dict "t" .Spend "title" "Spend" "note" ""}}{{end}}
|
||||
{{else}}<p class="empty">Nothing used in this window.</p>{{end}}
|
||||
{{end}}
|
||||
{{end}}
|
||||
@@ -1,20 +1,5 @@
|
||||
{{define "admin_quotas"}}<!doctype html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Quotas · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
||||
{{/* Links and forms work as plain navigation; htmx, when it loads, swaps only
|
||||
#quotas so opening a form or saving one does not jump the page. */}}
|
||||
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false}'>
|
||||
<script src="/_static/htmx-2.0.11.min.js" defer></script>
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>Quotas</h1>
|
||||
<p class="sub">What each organization, team and project may use · spend over the last {{.Since}}</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
|
||||
<div id="quotas" hx-boost="true" hx-target="#quotas" hx-select="#quotas"
|
||||
hx-swap="outerHTML" hx-push-url="false">
|
||||
{{/* The limits tab of the people page. */}}
|
||||
{{define "limits_body"}}
|
||||
|
||||
{{if not .BudgetsBind}}
|
||||
{{/* A budget is spend per period and spend comes from a price table. With no
|
||||
@@ -112,7 +97,7 @@ budgets bind.</p>
|
||||
up — hover it to see where. A value set here is in normal type.</p>
|
||||
{{else}}
|
||||
<p class="empty">Nothing to give a quota to yet. Make an organization on the
|
||||
<a href="/admin/users">people page</a> first.</p>
|
||||
<a href="/admin/users">Organizations & people</a> tab first.</p>
|
||||
{{end}}
|
||||
|
||||
<h2 id="people-limits">People</h2>
|
||||
@@ -198,7 +183,7 @@ asked. A model's owner is set on its card on the mirror page.</p>
|
||||
{{if .CapForm.Scope}}<a href="/admin/quotas#capacity">Cancel</a>{{end}}</div>
|
||||
</form>
|
||||
{{else}}<p class="empty">Nothing to give a capacity quota to yet. Make an organization on the
|
||||
<a href="/admin/users">people page</a> first.</p>{{end}}
|
||||
<a href="/admin/users">Organizations & people</a> tab first.</p>{{end}}
|
||||
{{end}}
|
||||
|
||||
<details class="explain">
|
||||
@@ -211,10 +196,8 @@ asked. A model's owner is set on its card on the mirror page.</p>
|
||||
putting every caller's throughput one click from an admin session is more
|
||||
than this page should do — <code>llmgw keys limits --scope key</code>.</p>
|
||||
</details>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
|
||||
</div></body></html>{{end}}
|
||||
|
||||
{{/* A limit as the table shows it: the row's own value, else the one it is
|
||||
held to from above, marked as inherited and saying from where. */}}
|
||||
@@ -224,6 +207,8 @@ asked. A model's owner is set on its card on the mirror page.</p>
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<input type="hidden" name="scope" value="{{.Scope}}">
|
||||
<input type="hidden" name="id" value="{{.ID}}">
|
||||
{{/* From a panel on the people tab: the answer comes back to that panel. */}}
|
||||
{{with .Sel}}<input type="hidden" name="tab" value="people"><input type="hidden" name="sel" value="{{.}}">{{end}}
|
||||
<p class="q-form-title"><strong>{{.Title}}</strong>. {{.Note}} A blank field means no limit of that kind.</p>
|
||||
<div class="field-row">
|
||||
<label>Requests / minute <input type="text" name="rpm" value="{{.F.RPM}}" placeholder="none"></label>
|
||||
@@ -239,7 +224,7 @@ asked. A model's owner is set on its card on the mirror page.</p>
|
||||
<input type="text" name="models" value="{{.F.Models}}" placeholder="every model"></label>
|
||||
<div class="q-form-actions">
|
||||
<button type="submit">Save</button>
|
||||
<a href="/admin/quotas">Cancel</a>
|
||||
{{if not .Sel}}<a href="/admin/quotas">Cancel</a>{{end}}
|
||||
<button type="submit" name="clear" value="1" class="link danger"
|
||||
{{if not .F.HasAny}}disabled{{end}}>Remove these limits</button>
|
||||
</div>
|
||||
|
||||
@@ -1,13 +1,5 @@
|
||||
{{define "admin_usage"}}<!doctype html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Usage · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>Usage over time</h1>
|
||||
<p class="sub">Who is using the cluster, and when</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
{{/* The usage tab of the people page. */}}
|
||||
{{define "usage_body"}}
|
||||
|
||||
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
|
||||
|
||||
@@ -63,6 +55,21 @@
|
||||
<p class="empty">Nothing recorded in this window.</p>
|
||||
{{end}}
|
||||
|
||||
{{with .GPUHours}}
|
||||
<h2>GPU time, by owner</h2>
|
||||
{{/* What each owner's models held, whether or not anybody asked them
|
||||
anything: a model kept loaded holds its cards all the same, and that is
|
||||
the time a GPU quota is about. */}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" $.GPUHeld "title" "GPUs held, by owner" "note" "The cards held by the replicas of each owner's models: models whose owner is a part of the quota tree."}}
|
||||
</div>
|
||||
<table class="rows">
|
||||
<tr><th>Owner</th><th class="num">GPU-hours</th><th class="num">Held now</th></tr>
|
||||
{{range .}}<tr><td>{{.Owner}}</td><td class="num">{{printf "%.1f" .Hours}}</td><td class="num">{{printf "%.0f" .Now}}</td></tr>{{end}}
|
||||
</table>
|
||||
<p class="note">From the metrics store, which keeps its history for as long as it is set to (fifteen days by default); a model with no owner is counted under “no owner”.</p>
|
||||
{{end}}
|
||||
|
||||
{{/* The charts fold their tail; the table does not. Both are here because they
|
||||
answer different questions, and the one that is easy to get wrong is
|
||||
thinking the four coloured lines are all of it. */}}
|
||||
@@ -73,7 +80,6 @@ is every group, in the same order.</p>
|
||||
<p class="note">A person is not a level of the quota tree — one person holds
|
||||
keys in several projects — so grouping by person crosses the tree rather than
|
||||
naming a level of it. Quotas are written on the other three, on the
|
||||
<a href="/admin/quotas">quotas page</a>.</p>
|
||||
<a href="/admin/quotas">Limits</a> tab.</p>
|
||||
|
||||
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
|
||||
</div></body></html>{{end}}
|
||||
{{end}}
|
||||
@@ -1,21 +1,5 @@
|
||||
{{define "admin_users"}}<!doctype html>
|
||||
<html lang="en"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>People · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
||||
{{/* Forms here post and get the whole page back, which works with no script at
|
||||
all. htmx, when it loads, swaps only #people from that page instead of
|
||||
reloading it, so an open project stays open and the page does not jump. */}}
|
||||
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false}'>
|
||||
<script src="/_static/htmx-2.0.11.min.js" defer></script>
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>People</h1>
|
||||
<p class="sub">{{len .Pending}} waiting for approval · {{len .Accounts}} accounts · {{len .Projects}} projects</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
|
||||
<div id="people" hx-boost="true" hx-target="#people" hx-select="#people"
|
||||
hx-swap="outerHTML" hx-push-url="false">
|
||||
{{/* The first tab of the people page: the tree, and everyone in it. */}}
|
||||
{{define "people_body"}}
|
||||
|
||||
{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
|
||||
{{if .Message}}<p class="ok" role="status">{{.Message}}</p>{{end}}
|
||||
@@ -47,8 +31,9 @@
|
||||
|
||||
<h2>Organizations, teams and projects</h2>
|
||||
<p class="note">A person is in a project, a project in a team, a team in an
|
||||
organization{{if .HasQuotas}}, and <a href="/admin/quotas">quotas</a> can be set
|
||||
on any of them{{end}}. Open a project to see its people and keys.</p>
|
||||
organization{{if .HasQuotas}}, and <a href="/admin/quotas">limits</a> can be set
|
||||
on any of them{{end}}. Choose a name — or a person in the list below — to see
|
||||
its limits, capacity and use beside the tree; open a project for its people and keys.</p>
|
||||
|
||||
{{if not .Tree}}
|
||||
<p class="empty">No organizations yet. Create one, then a team in it, then a
|
||||
@@ -58,7 +43,7 @@ project in the team — an account can only be approved into a project.</p>
|
||||
{{range $org := .Tree}}
|
||||
<section class="org">
|
||||
<div class="node-head">
|
||||
<strong class="node-name">{{$org.Name}}</strong>
|
||||
<strong class="node-name"><a href="/admin/users?sel=org:{{$org.ID}}" class="sel-link{{if eq $.Sel (printf "org:%s" $org.ID)}} on{{end}}">{{$org.Name}}</a></strong>
|
||||
<span class="node-meta">{{len $org.Teams}} team{{if ne (len $org.Teams) 1}}s{{end}}</span>
|
||||
{{template "scope-members" (dict "Kind" "org" "ID" $org.ID "Name" $org.Name "D" $)}}
|
||||
<details class="edit"><summary>rename</summary>
|
||||
@@ -80,7 +65,7 @@ project in the team — an account can only be approved into a project.</p>
|
||||
{{range $team := $org.Teams}}
|
||||
<div class="team">
|
||||
<div class="node-head">
|
||||
<span class="node-name">{{$team.Name}}</span>
|
||||
<span class="node-name"><a href="/admin/users?sel=team:{{$team.ID}}" class="sel-link{{if eq $.Sel (printf "team:%s" $team.ID)}} on{{end}}">{{$team.Name}}</a></span>
|
||||
<span class="node-meta">{{len $team.Projects}} project{{if ne (len $team.Projects) 1}}s{{end}}</span>
|
||||
{{template "scope-members" (dict "Kind" "team" "ID" $team.ID "Name" $team.Name "D" $)}}
|
||||
<details class="edit"><summary>rename</summary>
|
||||
@@ -114,6 +99,7 @@ project in the team — an account can only be approved into a project.</p>
|
||||
<details class="project" id="p-{{$p.ID}}"{{if eq $.Open $p.ID}} open{{end}}>
|
||||
<summary>
|
||||
<span class="node-name">{{$p.Name}}</span>
|
||||
<a href="/admin/users?sel=project:{{$p.ID}}&open={{$p.ID}}" class="sel-link small{{if eq $.Sel (printf "project:%s" $p.ID)}} on{{end}}" title="Limits and use of {{$p.Name}}">limits & use</a>
|
||||
<span class="node-meta">{{$p.Users}} member{{if ne $p.Users 1}}s{{end}}
|
||||
· {{$p.ActiveKeys}} key{{if ne $p.ActiveKeys 1}}s{{end}}{{if gt $p.Keys $p.ActiveKeys}} ({{sub $p.Keys $p.ActiveKeys}} revoked){{end}}</span>
|
||||
</summary>
|
||||
@@ -222,42 +208,8 @@ and issuing a key for someone else. Each turns a stolen admin session into more
|
||||
access than it had, so they need a shell on the control plane —
|
||||
<code>llmgw users promote</code>, <code>llmgw users reset-password</code>,
|
||||
<code>llmgw keys create</code>.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
|
||||
</div>
|
||||
<script>
|
||||
// Search the account list. Outside #people so what was typed survives a swap,
|
||||
// and applied again after every swap.
|
||||
(function () {
|
||||
var box = document.createElement('input');
|
||||
box.type = 'search'; box.placeholder = 'Find a person by email, name or project';
|
||||
box.className = 'search'; box.setAttribute('aria-label', 'Find a person');
|
||||
function apply() {
|
||||
var q = box.value.trim().toLowerCase();
|
||||
document.querySelectorAll('#account-list tr[data-q]').forEach(function (tr) {
|
||||
tr.hidden = q !== '' && tr.dataset.q.indexOf(q) < 0;
|
||||
});
|
||||
}
|
||||
box.addEventListener('input', apply);
|
||||
function place() {
|
||||
var h = document.getElementById('accounts');
|
||||
if (h && box.parentNode !== h.parentNode) h.after(box);
|
||||
apply();
|
||||
}
|
||||
place();
|
||||
// Projects the reader had open stay open across a change.
|
||||
var open = [];
|
||||
document.addEventListener('htmx:beforeSwap', function () {
|
||||
open = Array.prototype.map.call(document.querySelectorAll('details.project[open]'), function (d) { return d.id; });
|
||||
});
|
||||
document.addEventListener('htmx:afterSettle', function () {
|
||||
open.forEach(function (id) { var d = document.getElementById(id); if (d) d.open = true; });
|
||||
place();
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body></html>{{end}}
|
||||
|
||||
{{/* The actions every account row carries: suspend or reinstate, remove. */}}
|
||||
{{define "state-actions"}}{{$u := .U}}{{$d := .D}}
|
||||
@@ -295,7 +247,7 @@ access than it had, so they need a shell on the control plane —
|
||||
they could be. */}}
|
||||
{{define "account-row"}}{{$u := .U}}{{$d := .D}}
|
||||
<tr data-q="{{lower $u.Email}} {{lower $u.Name}}{{range $u.Memberships}} {{lower .Path}}{{end}}">
|
||||
<td>{{$u.Email}}{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
|
||||
<td><a href="/admin/users?sel=user:{{$u.ID}}" class="sel-link{{if eq $d.Sel (printf "user:%s" $u.ID)}} on{{end}}">{{$u.Email}}</a>{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
|
||||
<td class="state state-{{$u.State}}">{{$u.State}}</td>
|
||||
<td class="where">
|
||||
{{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
|
||||
|
||||
@@ -29,7 +29,7 @@
|
||||
{{if .Models}}
|
||||
<div class="chat-bar">
|
||||
<label class="inline-label">Model
|
||||
<select id="model">{{range .Models}}<option value="{{.Name}}" data-simulated="{{if .Simulated}}1{{else}}0{{end}}" data-ondemand="{{if .OnDemand}}1{{else}}0{{end}}">{{.Name}}{{if .Simulated}} (simulated){{end}}{{if .OnDemand}} (not loaded){{end}}</option>{{end}}</select>
|
||||
<select id="model">{{range .Models}}<option value="{{.Name}}" data-simulated="{{if .Simulated}}1{{else}}0{{end}}" data-ondemand="{{if .OnDemand}}1{{else}}0{{end}}">{{.Name}}{{with .Runs}} · {{.}}{{end}}{{if .Simulated}} (simulated){{end}}{{if .OnDemand}} (not loaded){{end}}</option>{{end}}</select>
|
||||
</label>
|
||||
<button id="reset" type="button" class="link">New conversation</button>
|
||||
</div>
|
||||
@@ -301,10 +301,13 @@ access to it.</p>
|
||||
let busy = null;
|
||||
|
||||
// A line under a reply: how long it took and how fast it came.
|
||||
function stats(out, tokens, t0, tFirst, tEnd, stopped) {
|
||||
function stats(out, tokens, t0, tFirst, tEnd, stopped, hw) {
|
||||
const p = document.createElement("p");
|
||||
p.className = "note chat-stats";
|
||||
const parts = [];
|
||||
// What answered: a model with replicas on a card and on CPUs is served
|
||||
// by either, and the speed below means nothing without knowing which.
|
||||
if (hw) parts.push(hw === "cpu" ? "CPU" : "GPU " + hw);
|
||||
if (tokens) parts.push(tokens + " tokens");
|
||||
if (tokens && tFirst && tEnd > tFirst) parts.push((tokens / ((tEnd - tFirst) / 1000)).toFixed(1) + " tok/s");
|
||||
if (tFirst) parts.push("first token " + ((tFirst - t0) / 1000).toFixed(2) + " s");
|
||||
@@ -428,7 +431,7 @@ access to it.</p>
|
||||
note.textContent = "Cut off at the length limit — ask it to continue.";
|
||||
out.after(note);
|
||||
}
|
||||
stats(out, tokens || pieces, t0, tFirst, performance.now(), stopped);
|
||||
stats(out, tokens || pieces, t0, tFirst, performance.now(), stopped, resp.headers.get("x-llmgw-hardware"));
|
||||
// Once, at the end. Re-rendering every chunk would try to typeset half
|
||||
// an expression on each token and flicker its way through the reply.
|
||||
await renderReply(out, reply);
|
||||
|
||||
@@ -49,6 +49,36 @@
|
||||
</div>
|
||||
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
|
||||
|
||||
{{with .GPUMap}}
|
||||
<h2>GPUs <span class="opt">by
|
||||
{{range .Modes}}{{if eq .Key $.GPUMap.By}}<strong>{{.Label}}</strong>{{else}}<a href="?gpumap={{.Key}}">{{.Label}}</a>{{end}} {{end}}</span></h2>
|
||||
<div class="gpu-map">
|
||||
{{range .Rows}}<div class="gpu-row"><span class="gpu-node">{{.Node}}</span>{{range .Cells}}<span class="gpu-cell" style="{{.Style}}" title="{{.Title}}">{{.Label}}</span>{{end}}</div>
|
||||
{{end}}</div>
|
||||
<p class="note">{{.Legend}} A red outline is a card wearing out — XID errors, memory rows remapped or waiting for a reset, NVLink errors. The figure is the one coloured by; the rest of a card is under the pointer.</p>
|
||||
{{end}}
|
||||
{{with .Efficiency}}
|
||||
<h3>What each model gets from its cards <span class="opt">last 15 minutes</span></h3>
|
||||
{{/* Per card, because a model on eight cards making twice the tokens of one
|
||||
on two is the worse of the two; per watt-hour, because that is what the
|
||||
room is paying. */}}
|
||||
<table class="rows"><tr><th>Model</th><th class="num">GPUs</th><th class="num">Output tokens/s</th><th class="num">Per GPU</th><th class="num">Busy</th><th class="num">Draw</th><th class="num">Tokens per Wh</th></tr>
|
||||
{{range .}}<tr><td>{{.Model}}</td><td class="num">{{printf "%.0f" .GPUs}}</td><td class="num">{{printf "%.1f" .TokensPerSec}}</td>
|
||||
<td class="num">{{printf "%.1f" .PerGPU}}</td><td class="num">{{printf "%.0f" .Util}}%</td>
|
||||
<td class="num">{{if .Watts}}{{printf "%.0f" .Watts}} W{{else}}—{{end}}</td><td class="num">{{if .PerWh}}{{printf "%.0f" .PerWh}}{{else}}—{{end}}</td></tr>
|
||||
{{end}}</table>
|
||||
{{end}}
|
||||
{{with .IdleGPUs}}
|
||||
<h3>Cards held and doing nothing</h3>
|
||||
{{/* The first place to find room: replicas whose every card reads under 5%
|
||||
busy now. A model kept warm on purpose is here too; whether it should
|
||||
be is the operator's call. */}}
|
||||
<table class="rows"><tr><th>Model</th><th>Replica</th><th>Node</th><th class="num">GPUs</th><th class="num">Busiest card</th><th>Last request</th></tr>
|
||||
{{range .}}<tr><td>{{.Model}}</td><td class="clip"><code>{{.Replica}}</code></td><td><code>{{.Node}}</code></td>
|
||||
<td class="num">{{len .GPUs}}</td><td class="num">{{printf "%.0f" .MaxUtil}}%</td><td>{{if .Idle}}{{.Idle}} ago{{else}}<span class="note">not known</span>{{end}}</td></tr>
|
||||
{{end}}</table>
|
||||
{{end}}
|
||||
|
||||
<h2>Nodes</h2>
|
||||
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
|
||||
{{if .Nodes}}
|
||||
|
||||
@@ -8,17 +8,13 @@
|
||||
{{- end}}
|
||||
{{- if .Admin}}
|
||||
<a href="/admin"{{if eq .Current "cluster"}} aria-current="page"{{end}}>Cluster</a>
|
||||
{{- /* Limits and usage are tabs of the people page, not pages of their own. */}}
|
||||
<a href="/admin/users"{{if eq .Current "people"}} aria-current="page"{{end}}>People</a>
|
||||
{{- if .Quotas}}
|
||||
<a href="/admin/quotas"{{if eq .Current "quotas"}} aria-current="page"{{end}}>Quotas</a>
|
||||
<a href="/admin/usage"{{if eq .Current "usage"}} aria-current="page"{{end}}>Usage</a>
|
||||
{{- end}}
|
||||
{{- if .Mirror}}
|
||||
<a href="/admin/mirror"{{if eq .Current "mirror"}} aria-current="page"{{end}}>Mirror</a>
|
||||
{{- end}}
|
||||
{{- if .Org}}
|
||||
<a href="/org"{{if eq .Current "org"}} aria-current="page"{{end}}>Organizations</a>
|
||||
{{- end}}
|
||||
{{- /* An organization is a panel of the people page for an administrator;
|
||||
/org is the managers' view, and leads an administrator there. */}}
|
||||
<span class="menu-gap" aria-hidden="true"></span>
|
||||
{{- end}}
|
||||
<a href="/chat"{{if eq .Current "chat"}} aria-current="page"{{end}}>Chat</a>
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
|
||||
{{with .Load}}{{if .GPUs}}
|
||||
<h2>GPUs</h2>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by</th></tr>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by, and wear</th></tr>
|
||||
{{range .GPUs}}<tr>
|
||||
<td class="num">{{.Index}}</td>
|
||||
{{/* From the last sample; a dash where the driver or an older agent did
|
||||
@@ -46,7 +46,7 @@
|
||||
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .SMClockMHz}}{{printf "%.0f" .SMClockMHz}} MHz{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{.PCIeReplays}} PCIe retries</span>{{end}}{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{end}}{{end}}
|
||||
{{end}}
|
||||
|
||||
Reference in new issue
Block a user