{{define "dashboard"}} llmgw — cluster

Cluster

{{.NodeCount}} nodes · {{.ReadyReplicas}} of {{.TotalReplicas}} replicas ready · {{.ModelCount}} models

{{if .OnAdminListener}}{{else}}{{template "nav" .Menu}}{{end}}
{{/* The page refreshes this part in place rather than reloading: a reload every few seconds threw away the scroll position of anyone reading the bottom of it. Paused while the tab is hidden. Without script, the meta refresh in the head does the old thing. */}}
{{if .HasCharts}}
{{.Requests24h}}
requests · 24h
{{.RequestSpark}}
{{compact .Tokens24h}}
tokens · 24h
{{.TokenSpark}}
{{.Errors24h}}
errors · 24h
{{units .Spend24h}}
spend · 24h
{{end}}

Capacity

{{if .Summary.GPUs}} {{range .Summary.GPUs}}{{end}}
ClassIn useFreeFaultyUsage
{{.Class}} {{.Used}}/{{.Total}} {{.Free}} {{if .Unhealthy}}{{.Unhealthy}}{{else}}—{{end}} {{pct .Used .Total}}%
{{else if .CPU.Nodes}} {{/* No GPUs is not no capacity: a CPU fleet still has memory and disk, which is what decides what it can hold. */}}
{{.CPU.Ready}}/{{.CPU.Nodes}} nodes ready · CPU only
{{.CPU.MemoryGB}} GB memory
{{if .CPU.DiskTotal}}
{{.CPU.DiskFreeGB}} of {{.CPU.DiskTotal}} GB free for weights
{{end}}
{{else}}

No nodes have registered.

{{end}}

Nodes

{{if .Nodes}} {{range .Nodes}} {{/* A dash, not an empty bar, when the agent has not measured it lately: unknown is not idle. */}} {{/* Free out of total, because the first is meaningless without the second, and an em dash where nothing was reported: an agent that was never asked about its disk must not read as a full one. */}} {{if .Unhealthy}}{{end}} {{range .Provision}}{{if ne .State "ready"}}{{end}}{{end}} {{range .Warnings}}{{end}} {{end}}
NodeStateCPURAMGPUsDiskEnginesClassesFabricsModelsReplicasLast seen
{{.ID}}{{if .MgmtAddr}} {{.MgmtAddr}}{{end}} logs {{.State}}{{with .Load}}{{template "minibar" pctbar .CPUPercent}}{{printf "%.0f" .CPUPercent}}%{{if .CPUs}} of {{.CPUs}}{{end}}{{else}}—{{end}} {{with .Load}}{{template "minibar" pctbar .MemPercent}}{{bytes .MemUsed}} of {{bytes .MemTotal}}{{else}}{{if .MemoryGB}}{{.MemoryGB}}G{{else}}—{{end}}{{end}} {{if .GPUTotal}}{{.GPUFree}}/{{.GPUTotal}}{{else}}—{{end}}{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}} {{if .Engines}}{{join .Engines}}{{else}}none yet{{end}} {{join .Classes}} {{join .Fabrics}} {{if .Models}}{{len .Models}}{{else}}—{{end}} {{.Replicas}} {{or .LastSeenAgo "—"}}
{{join .Unhealthy}}
{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{.State}}: {{.Detail}}
{{.}}
{{else}}

No agents have registered.

{{end}}

Models

{{if .Models}} {{range .Models}} {{/* Under the row rather than in a column: a sentence in a table cell either wraps every other column into a stripe or gets cut off. */}} {{if .Description}}{{end}} {{end}}
ModelAliasesReadyWantedProfilesFallbacks
{{.Name}} {{join .Aliases}} {{.Ready}} {{.Min}}–{{.Max}} {{join .Profiles}} {{join .Fallbacks}}
{{.Description}}
{{else}}

The catalog is empty.

{{end}} {{if .Unplaced}}

Why nothing was placed

{{range .Unplaced}}

{{.Model}}

{{range .Rejections}}{{.Profile}} @ {{or .Pool "-"}}{{if .Node}}/{{.Node}}{{end}}: {{.Reason}}
{{end}}
{{end}} {{end}}

Replicas

{{if .Replicas}} {{range .Replicas}}{{if .Err}}{{end}} {{end}}
ReplicaModelStatePoolEndpointAgeNodes
{{.ID}} {{.Model}} {{.State}} {{or .Pool "—"}} {{if .Endpoint}}{{.Endpoint}}{{else}}—{{end}} {{duration .AgeSec}} {{range .Shards}}{{.Node}} {{end}}
{{.Err}}
{{else}}

No replicas.

{{end}} {{if .Summary.IdleReplicas}}

Holding hardware nothing is asking for

{{range .Summary.IdleReplicas}}{{end}}
ReplicaModelGPUsUnused for
{{.ID}} {{.Model}} {{.GPUs}} {{duration .IdleSec}}
{{end}} {{if .Summary.Fabrics}}

Fabrics

{{range .Summary.Fabrics}}{{end}}
FabricNodesFully freeWeights
{{.Fabric}} {{.Nodes}} {{.LargestFreeBlock}} {{if .NodesWithWeights}}{{.Models}} model(s) on {{.NodesWithWeights}}/{{.Nodes}} nodes{{else}}not reported{{end}}

Fully free counts whole unused nodes, which is what decides whether a replica spanning nodes can land: free cards on the wrong fabric are not capacity for a model that needs this one.

{{end}} {{with .Fleet}}

What the cluster is doing last {{.Window}}

{{if .Err}} {{/* Said outright. An empty panel reads as a healthy cluster with nothing happening, which is the opposite of not knowing. */}}

No history: {{.Err}}

{{else}}
{{template "tschart" dict "t" .Saturation "title" "Pool saturation, by model" "note" "The worse of queue depth and cache pressure. 1 is the limit measured as healthy, not the point of failure."}} {{template "tschart" dict "t" .Ready "title" "Replicas ready, by model" "note" "A model deliberately stopped has no line at all, which is why an on-demand catalogue looks quiet here."}}
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica" "note" "Requests accepted and not yet started. The first thing to move when a replica is falling behind."}} {{template "tschart" dict "t" .KV "title" "KV cache occupancy, by replica" "note" "How full the attention cache is. Long contexts fill it with an empty queue, which is why saturation reads both."}}
{{end}} {{end}}

Traffic — last 24 hours, hourly

{{if .HasCharts}}

Requests

{{.RequestChart}}

Tokens

{{.TokenChart}}

Mean latency

{{.LatencyChart}}
{{else if .AuthEnabled}}

No traffic recorded yet.

{{end}} {{if .HasPeople}}

People

{{range .Users}}{{end}}
PersonKeysRequestsTokensErrorsSpendLast seen
{{if .Email}}{{.Email}}{{else}}operator-issued keys{{end}} {{.Keys}} {{.Requests}} {{compact .Tokens}} {{if .Errors}}{{.Errors}}{{else}}—{{end}} {{units .CostMicros}} {{ago .LastSeen}}
{{end}} {{if .Models2}}

What is being run

{{range .Models2}}{{end}}
ModelRequestsTokensErrorsMean latency
{{.Model}} {{.Requests}} {{compact .Tokens}} {{if .Errors}}{{.Errors}}{{else}}—{{end}} {{latency .AvgLatency}}

Counted by the model that actually answered, which differs from the one requested whenever a fallback served it.

{{end}}

Usage — last 24 hours

{{if .AuthEnabled}} {{if .Usage}} {{range .Usage}}{{end}}
PersonThroughSpends againstRequestsErrorsPromptCompletionSpendLast
{{or .Email "—"}} {{if .Chat}}chat page{{else if .KeyName}}{{.KeyName}} {{.KeyPrefix}}{{else}}{{.KeyID}}{{end}} {{if .ProjectPath}}{{.ProjectPath}}{{else if .ProjectID}}{{.ProjectID}}{{else}}personal{{end}} {{.Requests}} {{if .Errors}}{{.Errors}}{{else}}0{{end}} {{compact .Prompt}} {{compact .Completion}} {{units .CostMicros}} {{ago .LastAt}}

Spend is in units, whatever a unit was configured to mean. Usage is written in batches, so the last few seconds may be missing.

{{else}}

No requests recorded in the last 24 hours.

{{end}} {{else}}

Authentication is off, so there is no identity to account against.

{{end}} {{if not .OnAdminListener}}

Metrics, pprof and the management API are on the admin listener, which is bound to loopback and not reachable from here. pprof can dump the process heap, where API keys sit in plaintext, which is why that port is not exposed: ssh -L 9090:127.0.0.1:9090 <control-plane>

{{end}}
{{end}} {{/* One chart: the SVG, its legend, and the sentence that says how to read it. A legend is always present for two lines or more — colour alone is not an encoding a reader can rely on — and a single line needs none, because the title already names it. */}} {{define "tschart"}}

{{.title}}

{{if .t.Empty}}

Nothing recorded yet.

{{else}} {{.t.SVG}} peak {{.t.Peak}} {{if gt (len .t.Legend) 1}}
{{range .t.Legend}}{{.Name}} {{.Last}}{{end}}
{{end}} {{end}}

{{.note}}

{{end}} {{/* A small horizontal bar for a percentage, coloured as it fills. */}} {{define "minibar"}}{{end}}