{{define "node"}} llmgw — {{.Node.ID}}

{{.Node.ID}}

{{.Node.State}} {{with .Node.OS}} · {{.}}{{end}}{{with .Node.MgmtAddr}} · {{.}}{{end}} · last seen {{or .Node.LastSeenAgo "never"}} ago · logs

{{if .OnAdminListener}}{{else}}{{template "nav" .Nav}}{{end}}
{{/* Refreshed in place, like the cluster page. */}}
{{with .Node}}
{{with .Load}}
{{printf "%.0f" .CPUPercent}}%
CPU{{if .CPUs}} · {{.CPUs}} cores{{end}}
{{template "minibar" pctbar .CPUPercent}}
{{bytes .MemUsed}}
RAM in use · of {{bytes .MemTotal}}
{{template "minibar" pctbar .MemPercent}}
{{printf "%.2f" .Load1}}
load · {{printf "%.2f" .Load5}} 5m · {{printf "%.2f" .Load15}} 15m
{{/* Only what this machine has sensors for: a VM has none, and a tile saying 0 °C would be a reading nobody took. */}} {{if or .CPUTempC .CPUPowerW}}
{{if .CPUTempC}}{{printf "%.0f" .CPUTempC}} °C{{else}}—{{end}}
CPU{{if .CPUPowerW}} · {{printf "%.0f" .CPUPowerW}} W{{end}}
{{end}} {{if .InletTempC}}
{{printf "%.1f" .InletTempC}} °C
inlet air
{{end}} {{else}}
—
CPU and RAM not measured lately
{{end}} {{if .DiskTotalGB}}
{{.DiskFreeGB}}G
disk free · of {{.DiskTotalGB}}G
{{end}} {{if .GPUTotal}}
{{.GPUFree}}/{{.GPUTotal}}
GPUs free
{{end}}
{{if .Warnings}}{{range .Warnings}}

{{.}}

{{end}}{{end}} {{if .Unhealthy}}

{{join .Unhealthy}}

{{end}} {{with .Load}}{{if .GPUs}}

GPUs

{{range .GPUs}} {{/* From the last sample; a dash where the driver or an older agent did not say, rather than a zero nobody measured. */}} {{end}}
GPUBusyMemory usedTemperatureMemory temp.PowerClockHeld back by
{{.Index}}{{if and .MemoryTotalMB (ge .UtilPercent 0.0)}}{{template "minibar" pctbar .UtilPercent}}{{printf "%.0f" .UtilPercent}}%{{if ge .MemUtilPercent 0.0}} memory {{printf "%.0f" .MemUtilPercent}}%{{end}}{{else}}—{{end}} {{if .MemoryTotalMB}}{{template "minibar" pctbar (percent .MemoryUsedMB .MemoryTotalMB)}}{{printf "%.1f" (gib .MemoryUsedMB)}} of {{printf "%.0f" (gib .MemoryTotalMB)}} GB{{else}}—{{end}} {{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}} {{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}} {{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}{{printf "%.0f" .PowerW}} W of {{printf "%.0f" .PowerLimitW}}{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}} {{if .SMClockMHz}}{{printf "%.0f" .SMClockMHz}} MHz{{else}}—{{end}} {{if .Throttle}}{{join .Throttle}}{{else}}nothing{{end}}{{if gt .ECCUncorrected 0.0}} {{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors{{end}}
{{end}}{{end}} {{end}} {{if or .Node.IBPorts .Node.Fabrics}}

Fabric

{{if .Node.Fabrics}}On {{join .Node.Fabrics}} · the map →{{else}}On no fabric: no port a declared fabric names can carry traffic, so no multi-node replica is placed here.{{end}}

{{if .Node.IBPorts}} {{range .Node.IBPorts}}{{end}}
AdapterStateRateFabricPhysically onErrors since bootSentReceived
{{.HCA}}/{{.Port}}{{if eq .LinkLayer "Ethernet"}} RoCE{{end}} {{.State}}{{with .PhysState}} {{.}}{{end}} {{if .RateGbps}}{{.RateGbps}} Gb/s{{else}}—{{end}} {{or .Fabric "—"}} {{if .Physical}}{{.Physical}}{{else if .SMError}}unknown{{else}}—{{end}} {{if .Errors}}{{range $k, $v := .Errors}}{{$k}} {{$v}} {{end}}{{else}}none{{end}} {{if .TxBytes}}{{bytes (u64 .TxBytes)}}{{else}}—{{end}} {{if .RxBytes}}{{bytes (u64 .RxBytes)}}{{else}}—{{end}}

Re-read on every heartbeat. A port that goes down takes its fabric off this node within seconds; one that comes back returns it. Errors are cumulative since the adapter came up: a number that grows is the one to look at.

{{end}} {{end}} {{if .Node.GPUs}}

GPUs and NVLink

{{with .Node.NVLinkDomain}}In NVLink domain {{.}}: tensor parallelism can span this node and the rest of its domain. {{else}}NVLink ends at this machine.{{end}} {{if or .Node.NVLinkDomain .Node.IMEX}} · IMEX {{if eq .Node.IMEX "active"}}active, {{.Node.IMEXChannels}} channel{{if ne .Node.IMEXChannels 1}}s{{end}}{{else}}{{or .Node.IMEX "not installed"}}{{end}}{{end}}

{{range .Node.GPUs}}{{end}} {{range $a := .Node.GPUs}} {{range $b := $.Node.GPUs}}{{if eq $a.Index $b.Index}}{{else if hasInt $a.NVLinkPeers $b.Index}}{{else}}{{end}}{{end}} {{end}}
{{.Index}}
GPU {{$a.Index}}NV·
{{range .Node.GPUs}}{{end}}
GPUNameComputeNUMANearest adapterMulti-node NVLink
{{.Index}}{{.Name}}{{.ComputeCapability}}{{.NUMA}} {{if .NearestNIC}}{{.NearestNIC}} {{.NICPath}}{{else}}—{{end}} {{if .Domain}}joined{{if and .Bandwidth (ne .Bandwidth "Full")}} {{.Bandwidth}} bandwidth{{end}}{{else if .FabricState}}{{.FabricState}}{{else}}—{{end}}

NV is a pair of cards joined by NVLink; a tensor-parallel group kept to NV pairs runs its all-reduces at NVLink speed, and placement picks it that way. The nearest adapter is where a card's share of traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS is across the processors.

{{end}}

What runs here

{{if .Engines}} {{range .Engines}}{{end}}
ModelReplicaStateCPUMemory
{{.Model}} {{.Replica}} {{.State}} {{template "minibar" pctbar .CPUShare}}{{printf "%.1f" .CPUCores}} cores {{if .MemoryBytes}}{{bytes .MemoryBytes}}{{else}}—{{end}}
{{else}}

No engines are placed on this node.

{{end}}

Engines: {{if .Node.Engines}}{{join .Node.Engines}}{{else}}none installed{{end}}{{if .Node.Models}} · {{len .Node.Models}} model file(s) on disk{{end}}.

{{if .HasHistory}}{{with .Charts}}

The last {{.Window}}

{{if .Err}}

No history: {{.Err}}

{{else}}
{{template "tschart" dict "t" .CPU "title" "CPU" "note" "All cores together, from the node's heartbeat."}} {{template "tschart" dict "t" .Memory "title" "RAM in use" "note" "Total less what the kernel could hand to a new process."}}
{{template "tschart" dict "t" .EngineCPU "title" "Cores busy, by model" "note" "Each engine's process group, the way top counts it."}} {{template "tschart" dict "t" .EngineMemory "title" "Engine memory, by model" "note" "Resident memory, including the weights mapped from disk. The kernel can take those pages back, so this can exceed RAM in use."}}
{{if $.Node.GPUTotal}}
{{template "tschart" dict "t" .GPUUtil "title" "GPU busy, by card" "note" "Of each sample period, the share a kernel was running. A card held by a replica and idle here is one nobody is asking."}} {{template "tschart" dict "t" .GPUMemory "title" "GPU memory used, by card" "note" "Framebuffer in use: the weights and the KV cache an engine reserved, and anything not llmgw's."}}
{{template "tschart" dict "t" .GPUTemp "title" "GPU temperature, °C" "note" "Throttling usually starts in the high 80s; see what holds a card back in the table above."}} {{template "tschart" dict "t" .GPUPower "title" "GPU power, W" "note" "Draw against the card's enforced limit; at the limit is sw_power_cap."}}
{{end}}
{{template "tschart" dict "t" .Load "title" "Load average" "note" "Above the core count means work is waiting for a CPU."}} {{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica" "note" "Requests accepted and not yet started, on the replicas here now."}}
{{end}} {{end}}{{else}}

No metrics store is configured, so there is no history here; monitoring.prometheus in the server configuration adds it.

{{end}}
{{end}}