{{define "dashboard"}} llmgw — cluster

Cluster

{{.NodeCount}} nodes · {{.ReadyReplicas}} of {{.TotalReplicas}} replicas ready · {{.ModelCount}} models

Nodes

{{if .Nodes}} {{range .Nodes}}{{if .Unhealthy}}{{end}} {{end}}
NodeStateGPUsClassesFabricsReplicasLast seenAgent
{{.ID}}{{if .MgmtAddr}} {{.MgmtAddr}}{{end}} {{.State}} {{.GPUFree}}/{{.GPUTotal}} {{join .Classes}} {{join .Fabrics}} {{.Replicas}} {{or .LastSeenAgo "—"}} {{or .AgentVersion "—"}}
{{join .Unhealthy}}
{{else}}

No agents have registered.

{{end}}

Models

{{if .Models}} {{range .Models}}{{end}}
ModelAliasesReadyWantedProfilesFallbacks
{{.Name}} {{join .Aliases}} {{.Ready}} {{.Min}}–{{.Max}} {{join .Profiles}} {{join .Fallbacks}}
{{else}}

The catalog is empty.

{{end}} {{if .Unplaced}}

Why nothing was placed

{{range .Unplaced}}

{{.Model}}

{{range .Rejections}}{{.Profile}} @ {{or .Pool "-"}}{{if .Node}}/{{.Node}}{{end}}: {{.Reason}}
{{end}}
{{end}} {{end}}

Replicas

{{if .Replicas}} {{range .Replicas}}{{if .Err}}{{end}} {{end}}
ReplicaModelStatePoolEndpointAgeNodes
{{.ID}} {{.Model}} {{.State}} {{or .Pool "—"}} {{if .Endpoint}}{{.Endpoint}}{{else}}—{{end}} {{duration .AgeSec}} {{range .Shards}}{{.Node}} {{end}}
{{.Err}}
{{else}}

No replicas.

{{end}}

Usage — last 24 hours

{{if .AuthEnabled}} {{if .Usage}} {{range .Usage}}{{end}}
KeyProjectRequestsErrorsPromptCompletionSpendLast
{{if .KeyName}}{{.KeyName}} {{.KeyPrefix}}{{else}}{{.KeyID}}{{end}} {{.ProjectID}} {{.Requests}} {{if .Errors}}{{.Errors}}{{else}}0{{end}} {{.Prompt}} {{.Completion}} {{units .CostMicros}} {{ago .LastAt}}

Spend is in units, whatever a unit was configured to mean. Usage is written in batches, so the last few seconds may be missing.

{{else}}

No requests recorded in the last 24 hours.

{{end}} {{else}}

Authentication is off, so there is no identity to account against.

{{end}}
{{end}}