{{.NodeCount}} nodes · {{.ReadyReplicas}} of {{.TotalReplicas}} replicas ready · {{.ModelCount}} models
| Class | In use | Free | Faulty | Usage |
|---|---|---|---|---|
{{.Class}} |
{{.Used}}/{{.Total}} | {{.Free}} | {{if .Unhealthy}}{{.Unhealthy}}{{else}}—{{end}} | {{pct .Used .Total}}% |
No GPUs have been reported.
{{end}} {{if .Summary.Fabrics}}| Fabric | Nodes | Fully free | Weights |
|---|---|---|---|
{{.Fabric}} |
{{.Nodes}} | {{.LargestFreeBlock}} | {{if .NodesWithWeights}}{{.Models}} model(s) on {{.NodesWithWeights}}/{{.Nodes}} nodes{{else}}not reported{{end}} |
Fully free counts whole unused nodes, which is what decides whether a replica spanning nodes can land: free cards on the wrong fabric are not capacity for a model that needs this one.
{{end}} {{if .Summary.IdleReplicas}}| Replica | Model | GPUs | Unused for |
|---|---|---|---|
{{.ID}} |
{{.Model}} | {{.GPUs}} | {{duration .IdleSec}} |
| Person | Keys | Requests | Tokens | Errors | Spend | Last seen |
|---|---|---|---|---|---|---|
| {{if .Email}}{{.Email}}{{else}}operator-issued keys{{end}} | {{.Keys}} | {{.Requests}} | {{compact .Tokens}} | {{if .Errors}}{{.Errors}}{{else}}—{{end}} | {{units .CostMicros}} | {{ago .LastSeen}} |
| Model | Requests | Tokens | Errors | Mean latency |
|---|---|---|---|---|
{{.Model}} |
{{.Requests}} | {{compact .Tokens}} | {{if .Errors}}{{.Errors}}{{else}}—{{end}} | {{printf "%.0f" .AvgLatency}} ms |
Counted by the model that actually answered, which differs from the one requested whenever a fallback served it.
{{end}}| Node | State | GPUs | RAM | Classes | Fabrics | Models | Replicas | Last seen |
|---|---|---|---|---|---|---|---|---|
{{.ID}}{{if .MgmtAddr}} {{.MgmtAddr}}{{end}} |
{{.State}} | {{.GPUFree}}/{{.GPUTotal}} | {{if .MemoryGB}}{{.MemoryGB}}G{{else}}—{{end}} | {{join .Classes}} | {{join .Fabrics}} | {{if .Models}}{{len .Models}}{{else}}—{{end}} | {{.Replicas}} | {{or .LastSeenAgo "—"}} |
| {{join .Unhealthy}} | ||||||||
No agents have registered.
{{end}}| Model | Aliases | Ready | Wanted | Profiles | Fallbacks |
|---|---|---|---|---|---|
{{.Name}} |
{{join .Aliases}} | {{.Ready}} | {{.Min}}–{{.Max}} | {{join .Profiles}} | {{join .Fallbacks}} |
The catalog is empty.
{{end}} {{if .Unplaced}}{{.Model}}
{{range .Rejections}}{{.Profile}} @ {{or .Pool "-"}}{{if .Node}}/{{.Node}}{{end}}: {{.Reason}}
{{end}}
{{end}}
{{end}}
| Replica | Model | State | Pool | Endpoint | Age | Nodes |
|---|---|---|---|---|---|---|
{{.ID}} |
{{.Model}} | {{.State}} | {{or .Pool "—"}} | {{if .Endpoint}}{{.Endpoint}}{{else}}—{{end}} |
{{duration .AgeSec}} | {{range .Shards}}{{.Node}} {{end}} |
| {{.Err}} | ||||||
No replicas.
{{end}}Requests
{{.RequestChart}}Tokens
{{.TokenChart}}Mean latency
{{.LatencyChart}}No traffic recorded yet.
{{end}}| Key | Project | Requests | Errors | Prompt | Completion | Spend | Last |
|---|---|---|---|---|---|---|---|
{{if .KeyName}}{{.KeyName}} {{.KeyPrefix}}{{else}}{{.KeyID}}{{end}} |
{{.ProjectID}} | {{.Requests}} | {{if .Errors}}{{.Errors}}{{else}}0{{end}} | {{.Prompt}} | {{.Completion}} | {{units .CostMicros}} | {{ago .LastAt}} |
Spend is in units, whatever a unit was configured to mean. Usage is written in batches, so the last few seconds may be missing.
{{else}}No requests recorded in the last 24 hours.
{{end}} {{else}}Authentication is off, so there is no identity to account against.
{{end}} {{if not .OnAdminListener}}Metrics, pprof and the management API are on the admin listener,
which is bound to loopback and not reachable from here. pprof can dump the
process heap, where API keys sit in plaintext, which is why that port is not
exposed:
ssh -L 9090:127.0.0.1:9090 <control-plane>