{{define "dashboard"}} llmgw — cluster

Cluster

{{.NodeCount}} nodes · {{.ReadyReplicas}} of {{.TotalReplicas}} replicas ready · {{.ModelCount}} models

Nodes

{{if .Nodes}} {{range .Nodes}}{{if .Unhealthy}}{{end}} {{end}}
NodeStateGPUsClassesFabricsReplicasLast seenAgent
{{.ID}}{{if .MgmtAddr}} {{.MgmtAddr}}{{end}} {{.State}} {{.GPUFree}}/{{.GPUTotal}} {{join .Classes}} {{join .Fabrics}} {{.Replicas}} {{or .LastSeenAgo "—"}} {{or .AgentVersion "—"}}
{{join .Unhealthy}}
{{else}}

No agents have registered.

{{end}}

Models

{{if .Models}} {{range .Models}}{{end}}
ModelAliasesReadyWantedProfilesFallbacks
{{.Name}} {{join .Aliases}} {{.Ready}} {{.Min}}–{{.Max}} {{join .Profiles}} {{join .Fallbacks}}
{{else}}

The catalog is empty.

{{end}} {{if .Unplaced}}

Why nothing was placed

{{range .Unplaced}}

{{.Model}}

{{range .Rejections}}{{.Profile}} @ {{or .Pool "-"}}{{if .Node}}/{{.Node}}{{end}}: {{.Reason}}
{{end}}
{{end}} {{end}}

Replicas

{{if .Replicas}} {{range .Replicas}}{{if .Err}}{{end}} {{end}}
ReplicaModelStatePoolEndpointAgeNodes
{{.ID}} {{.Model}} {{.State}} {{or .Pool "—"}} {{if .Endpoint}}{{.Endpoint}}{{else}}—{{end}} {{duration .AgeSec}} {{range .Shards}}{{.Node}} {{end}}
{{.Err}}
{{else}}

No replicas.

{{end}}

Traffic — last 24 hours, hourly

{{if .HasCharts}}

Requests

{{.RequestChart}}

Tokens

{{.TokenChart}}

Mean latency

{{.LatencyChart}}
{{else if .AuthEnabled}}

No traffic recorded yet.

{{end}}

Usage — last 24 hours

{{if .AuthEnabled}} {{if .Usage}} {{range .Usage}}{{end}}
KeyProjectRequestsErrorsPromptCompletionSpendLast
{{if .KeyName}}{{.KeyName}} {{.KeyPrefix}}{{else}}{{.KeyID}}{{end}} {{.ProjectID}} {{.Requests}} {{if .Errors}}{{.Errors}}{{else}}0{{end}} {{.Prompt}} {{.Completion}} {{units .CostMicros}} {{ago .LastAt}}

Spend is in units, whatever a unit was configured to mean. Usage is written in batches, so the last few seconds may be missing.

{{else}}

No requests recorded in the last 24 hours.

{{end}} {{else}}

Authentication is off, so there is no identity to account against.

{{end}} {{if not .OnAdminListener}}

Metrics, pprof and the management API are on the admin listener, which is bound to loopback and not reachable from here. pprof can dump the process heap, where API keys sit in plaintext, which is why that port is not exposed: ssh -L 9090:127.0.0.1:9090 <control-plane>

{{end}}
{{end}}