Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 febcf251e4 Send an administrator from /org to the people page's panel
Release / build (push) Successful in 1m16s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m56s
An organization's members, limits and use are a panel of the people page,
where they can also be changed, so the administrator's menu loses its
Organizations entry and /org redirects there. Managers keep /org.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 19:15:25 +02:00
Ilker ManapandClaude Opus 5.5 fffed716be Make people, limits and usage one page, with a panel per branch
Release / build (push) Successful in 1m14s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m57s
A team's members, quota and spend were on three pages that linked to each
other. They are three tabs of one page now, each at its old address, and
choosing anything in the tree opens its limits, capacity and use beside it,
editable in place.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 19:10:48 +02:00
Ilker ManapandClaude Opus 5.5 1b89d5cdea Say on the chat page whether a GPU or a CPU answered
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m54s
A model with replicas on a card and on CPUs answered at either speed with no
way to tell which. The picker names where each model runs, each reply names
the hardware that served it, and every answer carries x-llmgw-hardware.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 18:57:50 +02:00
Ilker ManapandClaude Opus 5.5 477d0bf19d Judge a card shared with another process every heartbeat, not once
Release / build (push) Successful in 1m14s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m50s
A card with something else on it when the agent started stayed out of
placement after that process let go of it, until the agent restarted. The
agent now sends what processes other than its own engines hold on each card,
and the control plane replaces the registration verdict with it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 18:46:23 +02:00
34 changed files with 1146 additions and 155 deletions

No files matched your search

+46
View File
@@ -75,6 +75,52 @@ and a figure the driver does not give is left out rather than shown as zero.
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
### People, limits and usage are one page
Managing a team meant its members on the people page, its quota on the quotas
page and its spend on the usage page, each linking to the others. They are now
one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
each at the address it had before, so links and bookmarks still land. The menu
has one entry for them, People.
Choosing an organization, team, project or person in the tree, or a person in
the account list, opens a panel beside it with everything about that one
thing: its own limits and what it is held to from above, spend against its
budget, its per-member limit, its capacity quota, and its requests, tokens
and spend over the last week, drawn. Its limits can be set or changed there,
and the answer comes back to the same panel.
The administrator's menu no longer has an Organizations entry beside it: `/org`
and `/org/<id>` send an administrator to that organization's panel here. The
page stays as it was for an organization's managers, who cannot see this one.
### The chat page says whether a GPU or a CPU answered
A model can have replicas on a card and on CPUs at once, and the two answer
at very different speeds; the chat page gave no way to tell which one a reply
came from. The picker now says where each model runs — `GPU`, `CPU` or
`GPU + CPU` — and the line under each reply names what answered it: `GPU
6000-24g` or `CPU`, beside its tokens a second.
Every answer through the gateway carries the same as `x-llmgw-hardware`:
`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
none.
### A card shared with something else is judged every heartbeat
A card with another process on it when its agent started was kept out of
placement, and stayed out — "5014 MiB already allocated by another process" —
after that process let go of it, until the agent was restarted. Found on a
node shared with ComfyUI. The other way round, a tenant arriving after
registration was never seen, and a card the agent's own engine held when the
agent restarted read as someone else's.
The agent now reads every card's memory per process on each heartbeat,
leaves out what its own engines hold, and sends the rest; the control plane
keeps a card out of placement on that reading, and lets it back in when it
falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
agent too old to send it keeps the reading from registration.
### A GPU placement no longer runs on the CPU unnoticed
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
+21 -5
View File
@@ -2336,9 +2336,17 @@ type Heartbeat struct {
GpuFabric []*GPUFabricState `protobuf:"bytes,16,rep,name=gpu_fabric,json=gpuFabric,proto3" json:"gpu_fabric,omitempty"`
Imex *IMEX `protobuf:"bytes,17,opt,name=imex,proto3" json:"imex,omitempty"`
// cpu_count as in Register, for a VM given more CPUs while it runs.
CpuCount int32 `protobuf:"varint,18,opt,name=cpu_count,json=cpuCount,proto3" json:"cpu_count,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
CpuCount int32 `protobuf:"varint,18,opt,name=cpu_count,json=cpuCount,proto3" json:"cpu_count,omitempty"`
// gpu_foreign_memory_mb, indexed by GPU, is what processes that are not
// this agent's shards hold on each card, read afresh every heartbeat. What
// keeps a card out of placement for another tenant is decided from this
// rather than from registration: a card that had something else on it when
// the agent started is not taken forever, and one that gains a tenant later
// is not handed out. Empty from an agent that does not measure it, or from
// one that could not this time.
GpuForeignMemoryMb []int32 `protobuf:"varint,19,rep,packed,name=gpu_foreign_memory_mb,json=gpuForeignMemoryMb,proto3" json:"gpu_foreign_memory_mb,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Heartbeat) Reset() {
@@ -2497,6 +2505,13 @@ func (x *Heartbeat) GetCpuCount() int32 {
return 0
}
func (x *Heartbeat) GetGpuForeignMemoryMb() []int32 {
if x != nil {
return x.GpuForeignMemoryMb
}
return nil
}
// Provision is the software a node should have: which engines, at which
// versions. Nothing more.
//
@@ -4260,7 +4275,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x06status\x18\x03 \x01(\tR\x06status\x12!\n" +
"\fcluster_uuid\x18\x04 \x01(\tR\vclusterUuid\x12\x1b\n" +
"\tclique_id\x18\x05 \x01(\tR\bcliqueId\x12\x1c\n" +
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xf6\x06\n" +
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xa9\a\n" +
"\tHeartbeat\x12\x17\n" +
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12+\n" +
"\x12gpu_memory_used_mb\x18\x02 \x03(\x05R\x0fgpuMemoryUsedMb\x12!\n" +
@@ -4281,7 +4296,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\n" +
"gpu_fabric\x18\x10 \x03(\v2\x18.llmgw.v1.GPUFabricStateR\tgpuFabric\x12\"\n" +
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12\x1b\n" +
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x1aA\n" +
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x121\n" +
"\x15gpu_foreign_memory_mb\x18\x13 \x03(\x05R\x12gpuForeignMemoryMb\x1aA\n" +
"\x13EngineVersionsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
+9
View File
@@ -474,6 +474,15 @@ message Heartbeat {
IMEX imex = 17;
// cpu_count as in Register, for a VM given more CPUs while it runs.
int32 cpu_count = 18;
// gpu_foreign_memory_mb, indexed by GPU, is what processes that are not
// this agent's shards hold on each card, read afresh every heartbeat. What
// keeps a card out of placement for another tenant is decided from this
// rather than from registration: a card that had something else on it when
// the agent started is not taken forever, and one that gains a tenant later
// is not handed out. Empty from an agent that does not measure it, or from
// one that could not this time.
repeated int32 gpu_foreign_memory_mb = 19;
}
// Provision is the software a node should have: which engines, at which
+1 -1
View File
@@ -1271,7 +1271,7 @@ requests. On a slow model, set `timeouts.generation` higher.
can see each other, seeding between them is the obvious next thing to build.
- **A budget without a price never binds.** Set `default_price` or `pricing`
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
displays rather than something that stops anything. The Limits tab says so
where it can, which is not the same as being told before you rely on it.
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
no build from 0.20 on has Volta kernels, and the node picks the newest that
+16 -6
View File
@@ -191,7 +191,8 @@ project or person, the teams and projects with their limits and spend, and every
member. Only what was spent in that organization's projects is on it; what its
people do elsewhere is not. The page is read-only — a manager is not an
administrator of the cluster, and limits and memberships stay with those who
are.
are. An administrator following `/org` lands on the same organization's panel
on the people page instead, where all of that can be changed.
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
will no longer move a team or project that already exists under another parent;
@@ -259,7 +260,7 @@ Allowlists compose by intersection: a key cannot add a model its team withheld.
A model's canonical name and any of its aliases name the same thing, so either
form works.
Budgets are given in whole units, as the quotas page shows them, and what a unit
Budgets are given in whole units, as the Limits tab shows them, and what a unit
means is yours to decide. Self-hosted models have no per-token price, so it can
be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere —
a price of 1 000 000 in `auth.pricing`, which is in micro-units — and a budget
@@ -409,10 +410,19 @@ page exists.
opening onto its members and its keys. From there an admin approves people
(into a project or on their own), adds them to and takes them out of
organizations, teams and projects, suspends and removes them, revokes keys, and
adds, renames, moves and removes organizations, teams and projects. `/admin/quotas`
sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the
bottom. The page works as plain forms; with scripts on, a change updates it in
place and the projects you had open stay open.
adds, renames, moves and removes organizations, teams and projects. A
searchable list of every account is at the bottom. Choosing any organization,
team, project or person opens a panel beside the tree with its limits, its
capacity quota and its use over the last week, and its limits can be changed
there.
The same page has two more tabs, each at the address it had as a page of its
own: **Limits** (`/admin/quotas`) is every limit in the tree at once — per
branch, per member of any part of it, and per person — with the capacity quotas;
**Usage** (`/admin/usage`) is the cluster's use over time by organization, team,
project or person, and the GPU time each owner held. The page works as plain
forms; with scripts on, a change updates it in place and the projects you had
open stay open.
Three things are deliberately not there:
+18 -2
View File
@@ -591,8 +591,24 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
}
a.mu.Unlock()
for _, g := range inv.GPUs {
hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
// Measured now, not carried from registration: a card is shared with
// whatever else runs on the machine, and that comes and goes while the
// agent runs. The registration figure stays only for a node where
// nvidia-smi cannot be asked.
if len(inv.GPUs) > 0 {
var shards []int
for _, st := range a.sup.Statuses() {
if st.PID > 0 {
shards = append(shards, st.PID)
}
}
hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = a.foreignGPUMemory(context.Background(), shards)
}
if len(hb.GpuMemoryUsedMb) != len(inv.GPUs) {
hb.GpuForeignMemoryMb, hb.GpuMemoryUsedMb = nil, nil
for _, g := range inv.GPUs {
hb.GpuMemoryUsedMb = append(hb.GpuMemoryUsedMb, int32(g.MemoryUsedMB))
}
}
// Rescanned here rather than reused from registration. Disk is not a fact
+27 -5
View File
@@ -106,9 +106,27 @@ func parseSMI(out string) ([]GPU, error) {
return gpus, nil
}
// residentMemoryToleranceMB is how much allocated memory is written off as
// ForeignMemoryToleranceMB is how much allocated memory is written off as
// driver overhead rather than a foreign process.
const residentMemoryToleranceMB = 512
const ForeignMemoryToleranceMB = 512
// foreignMemorySuffix ends the problem a card with another tenant on it is
// kept out of placement for. One phrase, so that the control plane can find
// it again and drop it when the tenant leaves.
const foreignMemorySuffix = " MiB already allocated by another process"
// ForeignMemoryProblem is the problem for mb of memory held by something
// that is not one of this agent's engines, or "" when that is too little to
// count.
func ForeignMemoryProblem(mb int) string {
if mb <= ForeignMemoryToleranceMB {
return ""
}
return fmt.Sprintf("%d%s", mb, foreignMemorySuffix)
}
// IsForeignMemoryProblem reports whether a problem is ForeignMemoryProblem's.
func IsForeignMemoryProblem(p string) bool { return strings.HasSuffix(p, foreignMemorySuffix) }
// assessHealth flags conditions that should keep a GPU out of placement.
func assessHealth(g *GPU) {
@@ -119,8 +137,12 @@ func assessHealth(g *GPU) {
// Memory already in use means something else is on the card. Starting an
// engine there would either fail to allocate or quietly evict the other
// process's throughput.
if g.MemoryUsedMB > residentMemoryToleranceMB {
g.Health.Problems = append(g.Health.Problems,
fmt.Sprintf("%d MiB already allocated by another process", g.MemoryUsedMB))
//
// Only a first reading: this runs before the agent knows which processes
// are its own, so an engine still running from before a restart reads as
// foreign here. Every heartbeat measures it again, without the agent's
// own engines, and the control plane replaces this with that.
if p := ForeignMemoryProblem(g.MemoryUsedMB); p != "" {
g.Health.Problems = append(g.Health.Problems, p)
}
}
+61 -5
View File
@@ -34,13 +34,47 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
return parseGPUProcesses(out)
}
// GPUMemory is one card's memory as nvidia-smi reports it at a moment: what
// is allocated on it, and which processes hold how much of that.
type GPUMemory struct {
UsedMB int
Procs []GPUProcess
}
// GPUProcess is one process holding memory on a card.
type GPUProcess struct {
PID int
UsedMB int
}
// GPUMemoryByProcess reports each GPU's memory, by index, with the processes
// holding it — from the same report as GPUProcesses, so the two never
// disagree about who is on a card.
//
// A node with no nvidia-smi returns nil and no error, as discovery does.
func GPUMemoryByProcess(ctx context.Context, binary string) ([]GPUMemory, error) {
if binary == "" {
if _, err := exec.LookPath("nvidia-smi"); err != nil {
return nil, nil
}
binary = "nvidia-smi"
}
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
if err != nil {
return nil, fmt.Errorf("running %s: %w", binary, err)
}
return parseGPUMemory(out)
}
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
// index order, and the processes on it.
// index order, its memory and the processes on it.
type smiLog struct {
GPUs []struct {
Used string `xml:"fb_memory_usage>used"`
Processes []struct {
PID string `xml:"pid"`
Type string `xml:"type"`
PID string `xml:"pid"`
Type string `xml:"type"`
UsedMB string `xml:"used_memory"`
} `xml:"processes>process_info"`
} `xml:"gpu"`
}
@@ -48,17 +82,39 @@ type smiLog struct {
// parseGPUProcesses reads the report. GPUs are listed in index order, which
// is nvidia-smi's own numbering — the one placement assigns by.
func parseGPUProcesses(report []byte) (map[int][]int, error) {
mem, err := parseGPUMemory(report)
if err != nil {
return nil, err
}
out := map[int][]int{}
for i, g := range mem {
for _, p := range g.Procs {
out[i] = append(out[i], p.PID)
}
}
return out, nil
}
// parseGPUMemory reads the report into each GPU's memory and its holders.
func parseGPUMemory(report []byte) ([]GPUMemory, error) {
var log smiLog
if err := xml.Unmarshal(report, &log); err != nil {
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
}
out := map[int][]int{}
out := make([]GPUMemory, len(log.GPUs))
for i, g := range log.GPUs {
out[i].UsedMB = mib(g.Used)
for _, p := range g.Processes {
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
out[i] = append(out[i], pid)
out[i].Procs = append(out[i].Procs, GPUProcess{PID: pid, UsedMB: mib(p.UsedMB)})
}
}
}
return out, nil
}
// mib reads "4932 MiB"; anything else, "N/A" included, is zero.
func mib(s string) int {
n, _ := strconv.Atoi(strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(s), "MiB")))
return n
}
+40
View File
@@ -36,3 +36,43 @@ func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
t.Error("a report that does not parse was read as no processes")
}
}
func TestGPUMemoryIsReadPerProcess(t *testing.T) {
report := `<?xml version="1.0" ?>
<nvidia_smi_log>
<gpu id="00000000:01:00.0"><product_name>Quadro RTX 6000</product_name>
<fb_memory_usage><total>24576 MiB</total><used>14514 MiB</used><free>10062 MiB</free></fb_memory_usage>
<processes>
<process_info><pid>71315</pid><type>C</type><used_memory>4932 MiB</used_memory></process_info>
<process_info><pid>4100</pid><type>C</type><used_memory>9500 MiB</used_memory></process_info>
</processes>
</gpu>
<gpu id="00000000:02:00.0">
<fb_memory_usage><used>N/A</used></fb_memory_usage>
<processes></processes>
</gpu>
</nvidia_smi_log>`
got, err := parseGPUMemory([]byte(report))
if err != nil {
t.Fatal(err)
}
want := []GPUMemory{
{UsedMB: 14514, Procs: []GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 4100, UsedMB: 9500}}},
{},
}
if !reflect.DeepEqual(got, want) {
t.Errorf("got %+v, want %+v", got, want)
}
}
func TestForeignMemoryBelowTheToleranceIsNoProblem(t *testing.T) {
// A CUDA context that has let go of its model still holds a few hundred
// MiB: ComfyUI after /free held 324.
if p := ForeignMemoryProblem(330); p != "" {
t.Errorf("330 MiB is a problem: %q", p)
}
p := ForeignMemoryProblem(5014)
if !IsForeignMemoryProblem(p) || IsForeignMemoryProblem("2 uncorrected ECC errors") {
t.Errorf("%q is not recognised as the foreign memory problem, or ECC is", p)
}
}
+39
View File
@@ -16,6 +16,7 @@ import (
// nvidia-smi and /proc.
var (
gpuProcesses = discovery.GPUProcesses
gpuMemory = discovery.GPUMemoryByProcess
processGroup = pgidOf
// gpuCheckWait is how long a shard has, once it answers, to show up on
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
@@ -75,6 +76,44 @@ func (a *Agent) onItsGPUs(ctx context.Context, eng engines.Engine, spec engines.
"or run `llmgw doctor` on this node", joinIndices(missing), seen)
}
// foreignGPUMemory is what each GPU holds for anything other than this
// agent's shards, by index, and each GPU's memory in use. Nil when nvidia-smi
// cannot be asked, which is no reading rather than an empty card.
//
// Everything allocated less what the shards' processes hold, rather than the
// sum of the other processes: a process in another container's PID namespace
// holds memory nvidia-smi lists under no PID this node can see, and it is
// still not memory placement can hand out.
func (a *Agent) foreignGPUMemory(ctx context.Context, shards []int) (foreign, used []int32) {
mem, err := gpuMemory(ctx, a.cfg.Discovery.NvidiaSMI)
if err != nil || mem == nil {
return nil, nil
}
groups := make([]int, len(shards))
for i, pid := range shards {
groups[i] = processGroup(pid)
}
ours := func(p int) bool {
for i, pid := range shards {
if holds([]int{p}, pid, groups[i]) {
return true
}
}
return false
}
for _, g := range mem {
f := g.UsedMB
for _, p := range g.Procs {
if ours(p.PID) {
f -= p.UsedMB
}
}
foreign = append(foreign, int32(max(f, 0)))
used = append(used, int32(g.UsedMB))
}
return foreign, used
}
// holds reports whether any of pids is the shard's process or in its group:
// an engine that computes in a child process is still the shard.
func holds(pids []int, pid, group int) bool {
+33
View File
@@ -3,10 +3,12 @@ package agent
import (
"context"
"log/slog"
"slices"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
)
@@ -89,3 +91,34 @@ func TestTheCheckWaitsForTheShardToShowUpOnItsGPU(t *testing.T) {
t.Fatalf("nvidia-smi lagging by a second failed the shard: %v", err)
}
}
func TestForeignGPUMemoryLeavesOutTheAgentsOwnShards(t *testing.T) {
// cuda01: ComfyUI and one of our engines on the same card. Only ComfyUI's
// share — and whatever no listed process accounts for — is foreign.
oldMem, oldGroup := gpuMemory, processGroup
gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) {
return []discovery.GPUMemory{
{UsedMB: 15000, Procs: []discovery.GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 512, UsedMB: 9500}}},
{UsedMB: 300},
}, nil
}
processGroup = func(pid int) int {
if pid >= 500 && pid < 600 {
return 500
}
return pid
}
t.Cleanup(func() { gpuMemory, processGroup = oldMem, oldGroup })
a := &Agent{log: slog.Default()}
foreign, used := a.foreignGPUMemory(context.Background(), []int{500})
if !slices.Equal(foreign, []int32{5500, 300}) || !slices.Equal(used, []int32{15000, 300}) {
t.Errorf("foreign %v, used %v; want [5500 300] and [15000 300]", foreign, used)
}
// No nvidia-smi to ask is no reading, not an empty card.
gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) { return nil, nil }
if foreign, _ := a.foreignGPUMemory(context.Background(), nil); foreign != nil {
t.Errorf("no reading came back as %v", foreign)
}
}
@@ -283,3 +283,36 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
t.Errorf("the agent's refusal was not passed on: %v", err)
}
}
func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
// was kept out of placement. ComfyUI let go of it and the card stayed
// out, saying "5014 MiB already allocated by another process", until the
// agent was restarted.
s := newTestService()
nodeWith(s, inventory.Node{
ID: "n1", State: inventory.Ready,
GPUs: []discovery.GPU{{Index: 0, Health: discovery.GPUHealth{Problems: []string{
discovery.ForeignMemoryProblem(5014), "2 uncorrected ECC errors"}}}},
})
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{330}})
n, _ := s.inv.Get("n1")
if got := n.GPUs[0].Health.Problems; len(got) != 1 || got[0] != "2 uncorrected ECC errors" {
t.Fatalf("problems are %q: the tenant left, and only the ECC errors should remain", got)
}
// And the other way: a tenant arriving after registration.
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuForeignMemoryMb: []int32{12000}})
n, _ = s.inv.Get("n1")
if got := n.GPUs[0].Health.Problems; len(got) != 2 || got[1] != discovery.ForeignMemoryProblem(12000) {
t.Fatalf("problems are %q, want the new tenant's 12000 MiB", got)
}
// An agent that sends no reading changes nothing.
s.applyHeartbeat("n1", &llmgwv1.Heartbeat{NodeId: "n1", GpuMemoryUsedMb: []int32{100}})
n, _ = s.inv.Get("n1")
if len(n.GPUs[0].Health.Problems) != 2 {
t.Errorf("a heartbeat with no foreign reading cleared the card: %q", n.GPUs[0].Health.Problems)
}
}
+16 -1
View File
@@ -377,11 +377,12 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
// a reconnect.
func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
used := hb.GetGpuMemoryUsedMb()
foreign := hb.GetGpuForeignMemoryMb()
problems := hb.GetNewProblems()
storage := hb.GetStorage()
memMB := int(hb.GetMemoryTotalMb())
cpus := int(hb.GetCpuCount())
if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 &&
!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
@@ -397,6 +398,20 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
n.GPUs[i].MemoryUsedMB = int(used[i])
}
}
// Another tenant on a card, judged afresh: the reading at registration is
// replaced, both ways. Without this a card that had something on it when
// its agent started stayed out of placement after it left, and said so,
// until the agent was restarted. Only a reading for every card replaces
// it; an agent that sends none, or could not measure, leaves it standing.
if len(foreign) == len(n.GPUs) {
for i := range n.GPUs {
ps := slices.DeleteFunc(slices.Clone(n.GPUs[i].Health.Problems), discovery.IsForeignMemoryProblem)
if p := discovery.ForeignMemoryProblem(int(foreign[i])); p != "" {
ps = append(ps, p)
}
n.GPUs[i].Health.Problems = ps
}
}
if len(problems) > 0 {
// Problems are reported per node, not per GPU, so attach them to the
// node's first GPU rather than guessing which one is affected. Any
+13 -1
View File
@@ -255,6 +255,18 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
}
seen := map[string]struct{}{}
var out []web.ChatModel
// What each model's replicas compute on, across all of them: one model
// can run on a card and on CPUs at once.
hardware := map[string][]string{}
for _, s := range r.reg.Snapshots("") {
hw := hardwareOf(s.Pool)
for _, m := range s.Models {
name, _ := r.resolve(m)
if hw != "" && !slices.Contains(hardware[name], hw) {
hardware[name] = append(hardware[name], hw)
}
}
}
for _, s := range r.reg.Snapshots("") {
for _, m := range s.Models {
// Resolved rather than taken as sent: an alias and the name it
@@ -265,7 +277,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
}
seen[name] = struct{}{}
if r.permits(principal, name) {
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated()})
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
}
}
}
+50
View File
@@ -0,0 +1,50 @@
package router
import (
"net/http"
"net/http/httptest"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
func TestAReplySaysWhatHardwareAnsweredIt(t *testing.T) {
// qwen2.5-coder-14b had one replica on cuda01's card and one on a CPU
// node, and the chat page could not tell which had answered: the same
// model, ten times faster or slower, for no visible reason.
for pool, want := range map[string]string{
"none/6000-24g": "6000-24g",
"none/cpu": "cpu",
"": "",
} {
e := newFakeEngine(t, "m")
rt := newTestRouter(t, baseConfig())
rt.SetEndpoints(t.Context(), []endpoint.Info{
{ID: "r", URL: e.srv.URL, Models: []string{"m"}, Pool: pool, Weight: 1},
})
waitForEndpoint(t, rt)
rec := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil)
if rec.Code != http.StatusOK {
t.Fatalf("%q: status %d: %s", pool, rec.Code, rec.Body.String())
}
if got := rec.Header().Get(hardwareHeader); got != want {
t.Errorf("pool %q: hardware %q, want %q", pool, got, want)
}
}
}
func TestThePickerSaysWhereEachModelRuns(t *testing.T) {
gpu, cpu := newFakeEngine(t, "qwen2.5-coder-7b-instruct"), newFakeEngine(t, "qwen2.5-coder-7b-instruct")
rt, user := chatRouter(t, gpu.srv.URL)
rt.SetEndpoints(t.Context(), []endpoint.Info{
{ID: "g", URL: gpu.srv.URL, Models: []string{"qwen2.5-coder-7b-instruct"}, Pool: "none/6000-24g", Weight: 1},
{ID: "c", URL: cpu.srv.URL, Models: []string{"coder"}, Pool: "none/cpu", Weight: 1},
})
waitForEndpoint(t, rt)
got := rt.modelsFor(httptest.NewRequest("GET", "/chat", nil), user)
if len(got) != 1 || got[0].Runs() != "GPU + CPU" {
t.Fatalf("picker %+v; want one model running on GPU + CPU, its alias's replica included", got)
}
}
+25
View File
@@ -749,6 +749,16 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
if a, ok := r.adapter(req.Model); ok && a.base == model {
body = withAdapter(body, model, a.index, a.scale)
}
// What computes the answer, said before the proxy writes the headers.
// A model can have replicas on cards and on CPUs at once, and the two
// differ tenfold in speed: a reply that came slowly is otherwise
// indistinguishable from a model that is slow everywhere. Set again on
// a retry, which only happens before anything was written.
if hw := hardwareOf(dec.Endpoint.Pool); hw != "" {
w.Header().Set(hardwareHeader, hw)
} else {
w.Header().Del(hardwareHeader)
}
out, perr := r.px.Do(uctx, w, proxy.Request{
URL: dec.Endpoint.URL + req.Path,
APIKey: dec.Endpoint.APIKey,
@@ -791,6 +801,21 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
return last
}
// hardwareHeader says what the replica that answered computes on: "cpu", or
// the GPU class, such as "6000-24g".
const hardwareHeader = "x-llmgw-hardware"
// hardwareOf reads it from a pool, which placement names fabric/class —
// "none/cpu", "none/6000-24g", "ib-a/h100-80g". Empty for an endpoint
// configured by hand, which has no pool and says nothing about its hardware.
func hardwareOf(pool string) string {
i := strings.LastIndexByte(pool, '/')
if i < 0 || i == len(pool)-1 {
return ""
}
return pool[i+1:]
}
func without(eps []endpoint.Snapshot, id string) []endpoint.Snapshot {
return slices.DeleteFunc(slices.Clone(eps), func(e endpoint.Snapshot) bool { return e.ID == id })
}
+7
View File
@@ -88,6 +88,13 @@ type AccountPages struct {
// could run it. Both nil where the mirror page is read-only.
catalog CatalogAdmin
fleet Source
// The stores behind the people page's three tabs — the tree and its
// people, the limits, the usage — each set when its part is mounted, so
// the page draws the parts that exist.
peopleStore AccountAdmin
quotaStore QuotaAdmin
usageStore UsageAdmin
usagePriced Priced
}
// ServedModels supplies the names the profile page's example should use.
+259
View File
@@ -0,0 +1,259 @@
package web
import (
"context"
"net/http"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// The people page's tabs. One page, because people, the limits they are held
// to and what they used are three views of one tree: managing a team meant its
// members on one page, its quota on another and its spend on a third, each
// linking to the others. Each tab keeps the address it had as a page of its
// own, so links and bookmarks to /admin/quotas and /admin/usage still land.
const (
tabPeople = "people"
tabLimits = "limits"
tabUsage = "usage"
)
// peopleView is what a request asks the people page for.
type peopleView struct {
Tab string
Msg, Err string
// Open is the project shown expanded in the tree.
Open string
}
type peopleData struct {
Nav Nav
Tab string
// The tabs this server has, each built only when it is the one shown.
HasPeople, HasLimits, HasUsage bool
Users *adminUsersData
Quotas *adminQuotasData
Usage *adminUsageData
// Detail is the branch or person chosen in the tree, nil when none is.
Detail *peopleDetail
// Sel is what was chosen, "team:<id>" or "user:<id>", for links to keep.
Sel string
Pending int
Version string
Now string
}
// peopleDetail is one organization, team, project or person, with everything
// the three tabs say about it in one place.
type peopleDetail struct {
Kind store.Scope
ID string
// Name is what it is called, Path where it sits.
Name, Path string
// Limits is its own quota row, with what it is held to from above; nil
// for a person, or where limits are not mounted.
Limits *quotaRow
// Person is a person's own limit.
Person *personRow
// Memberships are where a person is.
Memberships []store.Membership
// Capacity is its capacity quota, nil when it has none.
Capacity *capacityRow
HasCapacity bool
BudgetsBind bool
// Its use over the last week, and the totals beside it.
HasUsage bool
Requests, Tokens, Spend TimeSeries
Totals usageRow
Priced bool
UsageWindow, QuotaWindow string
CSRF string
}
// renderPeople draws the people page on the tab asked for.
func (p *AccountPages) renderPeople(w http.ResponseWriter, r *http.Request, admin store.User, v peopleView) {
d := peopleData{
Nav: navFrom(r, "people"), Tab: v.Tab,
HasPeople: p.peopleStore != nil, HasLimits: p.quotaStore != nil, HasUsage: p.usageStore != nil,
Sel: r.FormValue("sel"),
Version: p.version, Now: time.Now().UTC().Format(time.RFC3339),
}
switch {
case d.Tab == tabPeople && d.HasPeople:
u := p.buildUsers(r, p.peopleStore, admin, v.Msg, v.Err, v.Open)
d.Users, d.Pending = &u, len(u.Pending)
if d.Sel != "" {
d.Detail = p.detail(r, d.Sel, &u)
}
case d.Tab == tabLimits && d.HasLimits:
q := p.buildQuotas(r, p.quotaStore, v.Msg, v.Err)
d.Quotas = &q
case d.Tab == tabUsage && d.HasUsage:
u := p.buildUsage(r, p.usageStore, p.usagePriced)
d.Usage = &u
default:
// The tab asked for is not mounted here: the first one that is.
switch {
case d.HasPeople:
v.Tab = tabPeople
case d.HasLimits:
v.Tab = tabLimits
case d.HasUsage:
v.Tab = tabUsage
default:
http.NotFound(w, r)
return
}
p.renderPeople(w, r, admin, v)
return
}
render(w, "admin_people", d)
}
// usageGrouping is the grouping a branch's or a person's use is read under.
func usageGrouping(kind store.Scope) (store.Grouping, bool) {
switch kind {
case store.ScopeOrg:
return store.GroupByOrg, true
case store.ScopeTeam:
return store.GroupByTeam, true
case store.ScopeProject:
return store.GroupByProject, true
case store.ScopeUser:
return store.GroupByUser, true
}
return "", false
}
// detail gathers what the page knows about one branch or person.
func (p *AccountPages) detail(r *http.Request, sel string, u *adminUsersData) *peopleDetail {
kind, id, ok := strings.Cut(sel, ":")
if !ok || id == "" {
return nil
}
ctx := r.Context()
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfFor(r),
UsageWindow: "last 7 days", QuotaWindow: "30 days"}
if !p.nameDetail(d, u) {
return nil
}
if p.quotaStore != nil {
d.BudgetsBind = p.priced != nil && p.priced.AnythingPriced()
p.limitsDetail(ctx, d)
}
if p.capacity != nil && d.Kind != store.ScopeUser {
d.HasCapacity = true
if views, err := p.capacity.QuotaViews(ctx); err == nil {
for _, v := range views {
if v.Scope == sel {
c := capacityRow{QuotaView: v,
GPUs: capCell(v.Holds.GPUs, v.Guarantee.GPUs, v.Limit.GPUs),
Cores: capCell(v.Holds.CPUCores, v.Guarantee.CPUCores, v.Limit.CPUCores),
Memory: capCell(v.Holds.MemoryGB, v.Guarantee.MemoryGB, v.Limit.MemoryGB)}
d.Capacity = &c
}
}
}
}
if p.usageStore != nil {
p.usageDetail(ctx, d)
}
return d
}
// nameDetail finds what was chosen in the tree or among the accounts, and
// says whether it is there at all.
func (p *AccountPages) nameDetail(d *peopleDetail, u *adminUsersData) bool {
if d.Kind == store.ScopeUser {
for _, a := range u.Accounts {
if a.ID == d.ID {
d.Name, d.Path, d.Memberships = a.Email, a.Name, a.Memberships
return true
}
}
return false
}
for _, o := range u.Tree {
if d.Kind == store.ScopeOrg && o.ID == d.ID {
d.Name, d.Path = o.Name, o.Name
return true
}
for _, t := range o.Teams {
if d.Kind == store.ScopeTeam && t.ID == d.ID {
d.Name, d.Path = t.Name, o.Name+" / "+t.Name
return true
}
for _, pr := range t.Projects {
if d.Kind == store.ScopeProject && pr.ID == d.ID {
d.Name, d.Path = pr.Name, o.Name+" / "+t.Name+" / "+pr.Name
return true
}
}
}
}
return false
}
// limitsDetail is the branch's own quota row, or the person's own limit.
func (p *AccountPages) limitsDetail(ctx context.Context, d *peopleDetail) {
s := p.quotaStore
limits, err := s.AllLimits(ctx)
if err != nil {
p.log.Warn("could not read the limits for a detail panel", "err", err)
return
}
since := time.Now().Add(-quotaWindow)
if d.Kind == store.ScopeUser {
for _, row := range p.personRows(ctx, s, limits, since, "") {
if row.ID == d.ID {
row := row
d.Person = &row
}
}
return
}
tree, err := s.Tree(ctx)
if err != nil {
return
}
for _, row := range quotaRows(tree, limits, p.spendByScope(ctx, s, since)) {
if row.Scope == d.Kind && row.ID == d.ID {
ms := store.MemberScope(row.Scope)
if l, ok := limits[store.LimitOwner{Scope: ms, ID: row.ID}.CounterKey()]; ok {
row.MemberRaw, row.Member = fieldsOf(l), describeLimits(l)
}
d.Limits = &row
return
}
}
}
// usageDetail is the branch's or person's use over the last week, drawn.
func (p *AccountPages) usageDetail(ctx context.Context, d *peopleDetail) {
g, ok := usageGrouping(d.Kind)
if !ok {
return
}
win := windowFor("7d")
rows, err := p.usageStore.UsageBucketsBy(ctx, g, time.Now().Add(-win.Span), win.Width)
if err != nil {
p.log.Warn("could not read usage for a detail panel", "sel", string(d.Kind)+":"+d.ID, "err", err)
return
}
d.HasUsage = true
d.Priced = p.usagePriced != nil && p.usagePriced.AnythingPriced()
for _, s := range rows {
if s.ID == d.ID {
var table []usageRow
d.Requests, d.Tokens, d.Spend, table = usageCharts([]store.UsageSeries{s})
d.Totals = table[0]
return
}
}
}
+87
View File
@@ -0,0 +1,87 @@
package web
import (
"io"
"log/slog"
"net/http"
"net/url"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// peoplePage mounts all three tabs, as the server does.
func peoplePage(t *testing.T, q *fakeQuotas, u *fakeScopeSeries) *http.ServeMux {
t.Helper()
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
mux := http.NewServeMux()
p.RegisterAdminUsers(mux, &fakeAdmin{orgs: q.orgs})
p.RegisterAdminQuotas(mux, q, pricedYes{})
p.RegisterAdminUsage(mux, u, pricedYes{})
return mux
}
func TestABranchChosenInTheTreeShowsItsLimitsAndUseTogether(t *testing.T) {
// People, their limits and their use were three pages: a team's members
// on one, its quota on another, its spend on a third.
q := oneTeam()
rpm := 60
q.limits["org:org-1"] = store.Limits{Scope: store.ScopeOrg, ScopeID: "org-1", RPM: &rpm}
u := &fakeScopeSeries{byGroup: map[store.Grouping][]store.UsageSeries{
store.GroupByOrg: {{Grouping: store.GroupByOrg, ID: "org-1", Name: "Scania", Requests: 1234,
Buckets: []store.Bucket{{Start: time.Now().Add(-time.Hour), Requests: 1234}}}},
}}
body := strings.Join(strings.Fields(
mirrorGet(t, peoplePage(t, q, u), "/admin/users?sel=org:org-1").Body.String()), " ")
for _, want := range []string{
`class="people-detail"`, "<h2>Scania</h2>",
"<dt>Requests/min</dt><dd>60</dd>", // its own limit
"1.2k requests", // its use over the week
`name="sel" value="org:org-1"`, // changing it comes back here
`aria-current="page">Organizations &amp; people`,
} {
if !strings.Contains(body, want) {
t.Errorf("the panel lacks %q", want)
}
}
if u.asked[len(u.asked)-1] != store.GroupByOrg {
t.Errorf("use was read by %v, not by organization", u.asked)
}
}
func TestAChangeMadeInAPanelComesBackToThePanel(t *testing.T) {
q := oneTeam()
body := mirrorPost(t, peoplePage(t, q, &fakeScopeSeries{}), "/admin/quotas", url.Values{
"csrf": {"tok"}, "scope": {"org"}, "id": {"org-1"}, "rpm": {"30"},
"tab": {"people"}, "sel": {"org:org-1"},
}).Body.String()
if len(q.put) != 1 || *q.put[0].RPM != 30 {
t.Fatalf("the limit was not saved: %+v", q.put)
}
if !strings.Contains(body, `class="people-detail"`) || !strings.Contains(body, "rpm 30") {
t.Error("saving from the panel did not come back to it with the result")
}
}
func TestEachOldAddressOpensItsTab(t *testing.T) {
mux := peoplePage(t, oneTeam(), &fakeScopeSeries{})
for path, tab := range map[string]string{
"/admin/users": "Organizations &amp; people", "/admin/quotas": "Limits", "/admin/usage": "Usage",
} {
body := mirrorGet(t, mux, path).Body.String()
if !strings.Contains(body, `aria-current="page">`+tab) {
t.Errorf("%s does not open the %s tab", path, tab)
}
}
}
func TestSomethingNotInTheTreeOpensNoPanel(t *testing.T) {
body := mirrorGet(t, peoplePage(t, oneTeam(), &fakeScopeSeries{}), "/admin/users?sel=org:gone").Body.String()
if strings.Contains(body, `class="people-detail"`) {
t.Error("a panel opened for an organization that does not exist")
}
}
+25 -11
View File
@@ -159,10 +159,11 @@ type adminQuotasData struct {
func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, priced Priced) {
p.hasQuotas = true
p.priced = priced
p.quotaStore = s
mux.HandleFunc("GET /admin/quotas", p.adminOnly(
func(w http.ResponseWriter, r *http.Request, _ store.User) {
p.renderQuotas(w, r, s, "", "")
func(w http.ResponseWriter, r *http.Request, admin store.User) {
p.renderPeople(w, r, admin, peopleView{Tab: tabLimits})
}))
mux.HandleFunc("POST /admin/quotas", p.adminOnly(
@@ -177,10 +178,10 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
p.log.Warn("a quota change was refused",
"admin", admin.Email, "scope", r.FormValue("scope"),
"id", r.FormValue("id"), "err", err)
p.renderQuotas(w, r, s, "", err.Error())
p.renderQuotas(w, r, admin, "", err.Error())
return
}
p.renderQuotas(w, r, s, msg, "")
p.renderQuotas(w, r, admin, msg, "")
}))
mux.HandleFunc("POST /admin/quotas/capacity", p.adminOnly(
@@ -194,11 +195,11 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
if err != nil {
p.log.Warn("a capacity quota change was refused",
"admin", admin.Email, "scope", r.FormValue("scope"), "err", err)
p.renderQuotas(w, r, s, "", err.Error())
p.renderQuotas(w, r, admin, "", err.Error())
return
}
p.log.Info("capacity quota changed", "admin", admin.Email, "what", msg)
p.renderQuotas(w, r, s, msg, "")
p.renderQuotas(w, r, admin, msg, "")
}))
}
@@ -270,10 +271,24 @@ func (p *AccountPages) applyQuota(r *http.Request, s QuotaAdmin, admin store.Use
return string(scope) + " " + id + ": " + now, nil
}
// renderQuotas answers a change to a limit: on the limits tab, or on the
// people tab when it was made from a branch's own panel there.
func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
s QuotaAdmin, msg, errMsg string,
admin store.User, msg, errMsg string,
) {
const window = 30 * 24 * time.Hour
tab := tabLimits
if r.FormValue("tab") == tabPeople {
tab = tabPeople
}
p.renderPeople(w, r, admin, peopleView{Tab: tab, Msg: msg, Err: errMsg})
}
// quotaWindow is how far back spend against a budget is counted.
const quotaWindow = 30 * 24 * time.Hour
// buildQuotas is every limit, as the limits tab shows them.
func (p *AccountPages) buildQuotas(r *http.Request, s QuotaAdmin, msg, errMsg string) adminQuotasData {
const window = quotaWindow
d := adminQuotasData{
Nav: navFrom(r, "quotas"),
Since: "30 days", Message: msg, Error: errMsg,
@@ -289,8 +304,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
if err != nil {
p.log.Error("could not read the limits", "err", err)
d.Error = "could not read the limits"
render(w, "admin_quotas", d)
return
return d
}
spend := p.spendByScope(r.Context(), s, time.Now().Add(-window))
@@ -314,7 +328,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
d.Rows[i].Editing = d.Editing ==
string(d.Rows[i].Scope)+":"+d.Rows[i].ID
}
render(w, "admin_quotas", d)
return d
}
// spendByScope is what each group has used in the window, keyed like a limit.
+8 -9
View File
@@ -99,15 +99,15 @@ type adminUsageData struct {
// answers "who is using the cluster, and when" — including per person, which is
// not a level quotas can be written on at all.
func (p *AccountPages) RegisterAdminUsage(mux *http.ServeMux, s UsageAdmin, priced Priced) {
mux.HandleFunc("GET /admin/usage", p.AdminPage(
func(w http.ResponseWriter, r *http.Request) {
p.renderUsage(w, r, s, priced)
p.usageStore, p.usagePriced = s, priced
mux.HandleFunc("GET /admin/usage", p.adminOnly(
func(w http.ResponseWriter, r *http.Request, admin store.User) {
p.renderPeople(w, r, admin, peopleView{Tab: tabUsage})
}))
}
func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
s UsageAdmin, priced Priced,
) {
// buildUsage is the cluster's use over time, as the usage tab shows it.
func (p *AccountPages) buildUsage(r *http.Request, s UsageAdmin, priced Priced) adminUsageData {
by := store.Grouping(r.URL.Query().Get("by"))
if !by.Valid() {
by = store.GroupByTeam
@@ -137,13 +137,12 @@ func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
if err != nil {
p.log.Error("could not read usage over time", "by", by, "err", err)
d.Error = "could not read usage"
render(w, "admin_usage", d)
return
return d
}
d.Requests, d.Tokens, d.Spend, d.Rows = usageCharts(rows)
d.GPUHeld, d.GPUHours = p.gpuHours(r.Context(), win)
render(w, "admin_usage", d)
return d
}
// gpuHoursRow is the GPU time one owner's models held over the window.
+12 -3
View File
@@ -87,6 +87,9 @@ type adminUsersData struct {
Projects []store.ProjectPath
Orgs []store.Org
Teams []store.TeamPath
// Sel is the branch or person whose panel is open, "team:<id>", so the
// tree can mark it.
Sel string
// Open is the project whose section is shown expanded: the one just acted
// on, so the result of a change is in view without the reader hunting for
// it. Scripted pages keep every open section open; this is for the rest.
@@ -115,6 +118,7 @@ type adminUsersData struct {
// shell is no hardship. Revoking a key is here, because it only ever takes
// access away.
func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
p.peopleStore = s
mux.HandleFunc("GET /admin/users", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
p.renderUsers(w, r, s, admin, "", "", r.URL.Query().Get("open"))
}))
@@ -480,12 +484,17 @@ func (p *AccountPages) adminAction(s AccountAdmin,
}
func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
s AccountAdmin, admin store.User, msg, errMsg, open string,
_ AccountAdmin, admin store.User, msg, errMsg, open string,
) {
p.renderPeople(w, r, admin, peopleView{Tab: tabPeople, Msg: msg, Err: errMsg, Open: open})
}
// buildUsers is the tree and its people, as the first tab shows them.
func (p *AccountPages) buildUsers(r *http.Request, s AccountAdmin, admin store.User, msg, errMsg, open string) adminUsersData {
ctx := r.Context()
d := adminUsersData{
Nav: navFrom(r, "people"),
Message: msg, Error: errMsg, Open: open, AdminID: admin.ID,
Message: msg, Error: errMsg, Open: open, AdminID: admin.ID, Sel: r.FormValue("sel"),
CSRF: csrfFor(r),
Version: p.version,
HasMirror: p.hasMirror,
@@ -558,7 +567,7 @@ func (p *AccountPages) renderUsers(w http.ResponseWriter, r *http.Request,
d.Scopes = append(d.Scopes, scopeOption{"project:" + pp.ID, pp.Path})
}
sort.Slice(d.Scopes, func(i, j int) bool { return d.Scopes[i].Label < d.Scopes[j].Label })
render(w, "admin_users", d)
return d
}
func keyState(k store.Key, now time.Time) string {
+27
View File
@@ -18,6 +18,33 @@ type ChatModel struct {
// offered anyway: the first message is what asks, so hiding it until it
// runs meant it could never be started from here.
OnDemand bool
// Hardware is what the model's running replicas compute on: "cpu", or a
// GPU class. Several when it runs on both, and empty when nothing runs
// or nothing said.
Hardware []string
}
// Runs names the hardware for the picker: "GPU", "CPU", or "GPU + CPU" for a
// model with replicas on both, where which one answers is the router's choice
// and each reply says which it was.
func (m ChatModel) Runs() string {
var gpu, cpu bool
for _, h := range m.Hardware {
if h == "cpu" {
cpu = true
} else {
gpu = true
}
}
switch {
case gpu && cpu:
return "GPU + CPU"
case gpu:
return "GPU"
case cpu:
return "CPU"
}
return ""
}
// ChatContext is one thing the chat page can spend against: a project, or the
+16
View File
@@ -185,3 +185,19 @@ func TestTheTruncationNoticeIsNotPutInsideTheReply(t *testing.T) {
t.Errorf("the reply is written as HTML other than from /chat/render (%d assignments)", n)
}
}
func TestChatModelRunsNamesTheHardware(t *testing.T) {
for _, c := range []struct {
hw []string
want string
}{
{nil, ""},
{[]string{"cpu"}, "CPU"},
{[]string{"6000-24g"}, "GPU"},
{[]string{"cpu", "6000-24g"}, "GPU + CPU"},
} {
if got := (ChatModel{Hardware: c.hw}).Runs(); got != c.want {
t.Errorf("%v: %q, want %q", c.hw, got, c.want)
}
}
}
+3 -2
View File
@@ -25,7 +25,8 @@ func TestEveryPageDrawsTheSameMenu(t *testing.T) {
p.RegisterAdminMirror(mux, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{}}})
pages := map[string]string{
"/admin/users": "People", "/admin/quotas": "Quotas", "/admin/mirror": "Mirror",
// Limits and usage are tabs of the people page, which they mark.
"/admin/users": "People", "/admin/quotas": "People", "/admin/mirror": "Mirror",
}
var first string
for path, current := range pages {
@@ -44,7 +45,7 @@ func TestEveryPageDrawsTheSameMenu(t *testing.T) {
t.Errorf("%s draws a different menu:\n%s\nfrom:\n%s", path, plain, first)
}
}
for _, link := range []string{"/admin", "/admin/users", "/admin/quotas", "/admin/usage",
for _, link := range []string{"/admin", "/admin/users",
"/admin/mirror", "/chat", "/profile", "/logout"} {
if !strings.Contains(first, `"`+link+`"`) {
t.Errorf("the menu lacks %s", link)
+23
View File
@@ -3,6 +3,7 @@ package web
import (
"context"
"net/http"
"net/url"
"sort"
"time"
@@ -70,6 +71,9 @@ type orgPageData struct {
func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced Priced) {
p.orgs = s
mux.HandleFunc("GET /org", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
if p.toPeople(w, r, user, "") {
return
}
orgs, err := p.viewableOrgs(r.Context(), s, user)
if err == nil && len(orgs) == 0 && user.IsAdmin {
// Said, not a 404: an administrator following the link on a
@@ -89,6 +93,9 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
}))
mux.HandleFunc("GET /org/{id}", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
id := r.PathValue("id")
if p.toPeople(w, r, user, id) {
return
}
if !user.IsAdmin {
ok, err := s.ManagesOrg(r.Context(), user.ID, id)
if err != nil || !ok {
@@ -102,6 +109,22 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
}))
}
// toPeople sends an administrator to the people page, which shows an
// organization's members, limits and use beside the whole tree and can change
// them; this page stays for the managers who may not see that one. It says
// whether it did.
func (p *AccountPages) toPeople(w http.ResponseWriter, r *http.Request, user store.User, orgID string) bool {
if !user.IsAdmin || p.peopleStore == nil {
return false
}
to := "/admin/users"
if orgID != "" {
to += "?sel=org:" + url.QueryEscape(orgID)
}
http.Redirect(w, r, to, http.StatusSeeOther)
return true
}
// viewableOrgs is every organization an administrator may look at, or the
// ones a person manages.
func (p *AccountPages) viewableOrgs(ctx context.Context, s OrgViewer, user store.User) ([]store.Org, error) {
+19
View File
@@ -124,3 +124,22 @@ func TestWithNoOrganizationsAnAdministratorIsToldSo(t *testing.T) {
t.Error("the menu links to organizations when there are none")
}
}
func TestAnAdministratorIsSentToThePeoplePageForAnOrganization(t *testing.T) {
// Its members, limits and use are a panel there, beside the whole tree
// and changeable; /org stays for the managers who cannot see that page.
p := NewAccountPages(fakeAccounts{user: store.User{ID: "a", IsAdmin: true}}, nil, nil, "v0",
slog.New(slog.NewTextHandler(io.Discard, nil)))
mux := http.NewServeMux()
p.RegisterAdminUsers(mux, &fakeAdmin{})
p.RegisterOrgPages(mux, fakeOrgs{}, nil)
for path, to := range map[string]string{"/org": "/admin/users", "/org/org-s": "/admin/users?sel=org:org-s"} {
w := mirrorGet(t, mux, path)
if w.Code != http.StatusSeeOther || w.Header().Get("Location") != to {
t.Errorf("%s: %d to %q, want %s", path, w.Code, w.Header().Get("Location"), to)
}
}
if menu := menuRe.FindString(mirrorGet(t, mux, "/admin/users").Body.String()); strings.Contains(menu, `href="/org"`) {
t.Error("the administrator's menu still offers /org beside People")
}
}
+33
View File
@@ -869,3 +869,36 @@ table.rows td.warn { color: var(--warn); cursor: help; }
.gpu-node { width: 7em; flex: none; font-size: 12px; color: var(--muted, #666); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.gpu-cell { width: 26px; height: 26px; flex: none; border: 1px solid rgba(0,0,0,.18); border-radius: 4px;
font-size: 10px; line-height: 26px; text-align: center; color: rgba(0,0,0,.7); cursor: default; }
/* The people page: one page in tabs, and a panel for the branch or person
chosen in the tree, beside it on a wide screen and above it on a narrow one. */
.page-tabs { display: flex; flex-wrap: wrap; gap: .25rem; border-bottom: 1px solid var(--line); margin: .25rem 0 1rem; }
.page-tabs a { padding: .45rem .8rem; color: var(--muted); text-decoration: none; border-bottom: 2px solid transparent; margin-bottom: -1px; }
.page-tabs a:hover { color: var(--fg); }
.page-tabs a[aria-current="page"] { color: var(--fg); border-bottom-color: var(--accent); font-weight: 600; }
.people-grid.with-detail { display: grid; gap: 1.25rem; grid-template-columns: minmax(0, 1fr) minmax(0, 24rem); align-items: start; }
.people-grid.with-detail .people-detail { grid-column: 2; grid-row: 1; position: sticky; top: .75rem; max-height: calc(100vh - 1.5rem); overflow-y: auto; }
.people-grid.with-detail .people-main { grid-column: 1; grid-row: 1; min-width: 0; }
@media (max-width: 960px) {
.people-grid.with-detail { grid-template-columns: minmax(0, 1fr); }
.people-grid.with-detail .people-detail { grid-column: 1; grid-row: 1; position: static; max-height: none; }
.people-grid.with-detail .people-main { grid-row: 2; }
}
.people-detail { background: var(--card); border: 1px solid var(--line); border-radius: 8px; padding: .9rem 1rem; }
.people-detail h2 { margin: .1rem 0 0; font-size: 1.15rem; }
.people-detail h3 { margin: 1rem 0 .4rem; font-size: .9rem; text-transform: uppercase; letter-spacing: .04em; color: var(--muted); }
.people-detail .chart-wrap { margin-bottom: .75rem; }
.people-detail .q-form .field-row { flex-wrap: wrap; }
.pd-head { position: relative; padding-right: 1.5rem; }
.pd-path, .pd-sub { margin: .2rem 0 0; font-size: .85rem; color: var(--muted); }
.pd-close { position: absolute; top: 0; right: 0; font-size: 1.3rem; line-height: 1; color: var(--muted); text-decoration: none; }
.pd-limits { display: grid; grid-template-columns: auto 1fr; gap: .25rem .9rem; margin: 0; font-size: .88rem; }
.pd-limits dt { color: var(--muted); }
.pd-limits dd { margin: 0; font-variant-numeric: tabular-nums; }
a.sel-link { color: inherit; text-decoration: none; border-bottom: 1px dotted var(--line); }
a.sel-link:hover { color: var(--accent); border-bottom-color: var(--accent); }
a.sel-link.on { color: var(--accent); border-bottom: 2px solid var(--accent); }
a.sel-link.small { font-size: .78rem; color: var(--muted); margin-left: .4rem; }
.people-detail h3.chart-title { text-transform: none; letter-spacing: 0; color: var(--fg); font-size: .85rem; margin: .2rem 0; }
.people-detail details.edit { display: inline-block; margin-right: 1rem; vertical-align: top; }
.people-detail details.edit[open] { display: block; margin: .5rem 0; }
+148
View File
@@ -0,0 +1,148 @@
{{/* People, their limits and their use: one page in three tabs, because all
three are about the same tree. Each tab keeps its old address. Forms and
links work as plain navigation; htmx, when it loads, swaps only #people,
so a change does not jump the page. */}}
{{define "admin_people"}}<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>People · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false}'>
<script src="/_static/htmx-2.0.11.min.js" defer></script>
</head><body><div class="wrap">
<header>
<h1>People</h1>
<p class="sub">Organizations, teams, projects and the people in them: what each may use, and what it used</p>
{{template "nav" .Nav}}
</header>
<div id="people" hx-boost="true" hx-target="#people" hx-select="#people"
hx-swap="outerHTML" hx-push-url="false">
<nav class="page-tabs" aria-label="People">
{{if .HasPeople}}<a href="/admin/users" hx-push-url="true"{{if eq .Tab "people"}} aria-current="page"{{end}}>Organizations &amp; people{{if .Pending}} <span class="tag">{{.Pending}} waiting</span>{{end}}</a>{{end}}
{{if .HasLimits}}<a href="/admin/quotas" hx-push-url="true"{{if eq .Tab "limits"}} aria-current="page"{{end}}>Limits</a>{{end}}
{{if .HasUsage}}<a href="/admin/usage" hx-push-url="true"{{if eq .Tab "usage"}} aria-current="page"{{end}}>Usage</a>{{end}}
</nav>
{{if .Users}}
<div class="people-grid{{if .Detail}} with-detail{{end}}">
{{with .Detail}}<aside class="people-detail" id="detail" aria-label="{{.Name}}">{{template "people-detail" .}}</aside>{{end}}
<div class="people-main">{{template "people_body" .Users}}</div>
</div>
{{else if .Quotas}}{{template "limits_body" .Quotas}}
{{else if .Usage}}{{template "usage_body" .Usage}}
{{end}}
</div>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
</div>
<script>
// Search the account list. Outside #people so what was typed survives a swap,
// and applied again after every swap.
(function () {
var box = document.createElement('input');
box.type = 'search'; box.placeholder = 'Find a person by email, name or project';
box.className = 'search'; box.setAttribute('aria-label', 'Find a person');
function apply() {
var q = box.value.trim().toLowerCase();
document.querySelectorAll('#account-list tr[data-q]').forEach(function (tr) {
tr.hidden = q !== '' && tr.dataset.q.indexOf(q) < 0;
});
}
box.addEventListener('input', apply);
function place() {
var h = document.getElementById('accounts');
if (h && box.parentNode !== h.parentNode) h.after(box);
apply();
}
place();
// Projects the reader had open stay open across a change.
var open = [];
document.addEventListener('htmx:beforeSwap', function () {
open = Array.prototype.map.call(document.querySelectorAll('details.project[open]'), function (d) { return d.id; });
});
document.addEventListener('htmx:afterSettle', function () {
open.forEach(function (id) { var d = document.getElementById(id); if (d) d.open = true; });
place();
});
})();
</script>
</body></html>{{end}}
{{/* One branch or person: its limits, its capacity and its use, together. */}}
{{define "people-detail"}}{{$d := .}}{{$sel := printf "%s:%s" .Kind .ID}}
<div class="pd-head">
<span class="q-kind">{{if eq (printf "%s" .Kind) "user"}}person{{else}}{{.Kind}}{{end}}</span>
<h2>{{.Name}}</h2>
{{if and .Path (ne .Path .Name)}}<p class="pd-path">{{.Path}}</p>{{end}}
<a href="/admin/users" class="pd-close" aria-label="Close">×</a>
</div>
{{with .Memberships}}<p class="pd-sub">In {{range $i, $m := .}}{{if $i}}, {{end}}<a href="/admin/users?sel={{$m.Scope}}:{{$m.ScopeID}}">{{$m.Path}}</a>{{end}}</p>
{{else}}{{if eq (printf "%s" $d.Kind) "user"}}<p class="pd-sub">On their own: in no organization, team or project.</p>{{end}}{{end}}
{{if or .Limits .Person}}
<h3>Limits</h3>
{{if not .BudgetsBind}}<p class="note">No model has a price, so budgets stay at zero; rate and token limits apply.</p>{{end}}
{{with .Limits}}
<dl class="pd-limits">
<dt>Requests/min</dt><dd>{{template "quota-value" dict "Own" .RPM "Up" .RPMUp "From" .RPMFrom}}</dd>
<dt>Tokens/min</dt><dd>{{template "quota-value" dict "Own" .TPM "Up" .TPMUp "From" .TPMFrom}}</dd>
<dt>Budget</dt><dd>{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}{{else}}{{template "quota-value" dict "Own" "" "Up" .BudgetUp "From" .BudgetFrom}}{{end}}</dd>
<dt>Spent, {{$d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{.Percent}}%</span>{{end}}</dd>
<dt>Models</dt><dd>{{if .Models}}only <code>{{.Models}}</code>{{else if .ModelsFrom}}limited by {{.ModelsFrom}}{{else}}every model{{end}}</dd>
{{if .Member}}<dt>Each member</dt><dd>{{.Member}}</dd>{{end}}
</dl>
{{if .Shadowed}}<p class="q-sub q-warn">{{.Shadowed}}</p>{{end}}
<details class="edit"><summary>{{if .HasAny}}change{{else}}set{{end}} limits</summary>
{{template "quota-form" (dict "Scope" (printf "%s" .Scope) "ID" .ID "CSRF" $d.CSRF "Sel" $sel
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
"Title" (printf "Quota for %s" .Crumb)
"Note" "Shared by everything beneath it.")}}
</details>
<details class="edit"><summary>limit each member</summary>
{{template "quota-form" (dict "Scope" (printf "%s-member" .Scope) "ID" .ID "F" .MemberRaw "CSRF" $d.CSRF "Sel" $sel
"Title" (printf "Each member of %s, separately" .Crumb)
"Note" "Every member gets this much here, counted apart from the others.")}}
</details>
{{end}}
{{with .Person}}
<dl class="pd-limits">
<dt>Requests/min</dt><dd>{{or .RPM "—"}}</dd>
<dt>Tokens/min</dt><dd>{{or .TPM "—"}}</dd>
<dt>Budget</dt><dd>{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}{{else}}—{{end}}</dd>
<dt>Spent, {{$d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{.Percent}}%</span>{{end}}</dd>
<dt>Models</dt><dd>{{if .Models}}only <code>{{.Models}}</code>{{else}}every model{{end}}</dd>
</dl>
<details class="edit"><summary>{{if .HasAny}}change{{else}}set{{end}} limits</summary>
{{template "quota-form" (dict "Scope" "user" "ID" .ID "CSRF" $d.CSRF "Sel" $sel
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
"Title" (printf "%s, everywhere" .Email)
"Note" "Across every key this person holds, every project they use and the chat page.")}}
</details>
{{end}}
{{end}}
{{if .HasCapacity}}
<h3>Capacity</h3>
{{with .Capacity}}
<dl class="pd-limits">
<dt>GPUs</dt><dd>{{or .GPUs "—"}}</dd>
<dt>Cores</dt><dd>{{or .Cores "—"}}</dd>
<dt>Memory GB</dt><dd>{{or .Memory "—"}}</dd>
</dl>
<p class="note">Held of guaranteed (at most the limit). <a href="/admin/quotas?cap={{$sel}}#capacity">change</a></p>
{{else}}<p class="note">No capacity quota: its models share the cluster by their class alone.
<a href="/admin/quotas?cap={{$sel}}#capacity">set one</a></p>{{end}}
{{end}}
{{if .HasUsage}}
<h3>Use, {{.UsageWindow}}</h3>
{{if .Totals.Name}}
<p class="pd-sub">{{compact .Totals.Requests}} requests · {{compact .Totals.Tokens}} tokens{{if .Priced}} · {{.Totals.Spend}} spent{{end}}{{if .Totals.Errors}} · {{.Totals.Errors}} errors{{end}}</p>
{{template "tschart" dict "t" .Requests "title" "Requests" "note" ""}}
{{template "tschart" dict "t" .Tokens "title" "Tokens" "note" ""}}
{{if .Priced}}{{template "tschart" dict "t" .Spend "title" "Spend" "note" ""}}{{end}}
{{else}}<p class="empty">Nothing used in this window.</p>{{end}}
{{end}}
{{end}}
+8 -23
View File
@@ -1,20 +1,5 @@
{{define "admin_quotas"}}<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Quotas · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
{{/* Links and forms work as plain navigation; htmx, when it loads, swaps only
#quotas so opening a form or saving one does not jump the page. */}}
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false}'>
<script src="/_static/htmx-2.0.11.min.js" defer></script>
</head><body><div class="wrap">
<header>
<h1>Quotas</h1>
<p class="sub">What each organization, team and project may use · spend over the last {{.Since}}</p>
{{template "nav" .Nav}}
</header>
<div id="quotas" hx-boost="true" hx-target="#quotas" hx-select="#quotas"
hx-swap="outerHTML" hx-push-url="false">
{{/* The limits tab of the people page. */}}
{{define "limits_body"}}
{{if not .BudgetsBind}}
{{/* A budget is spend per period and spend comes from a price table. With no
@@ -112,7 +97,7 @@ budgets bind.</p>
up — hover it to see where. A value set here is in normal type.</p>
{{else}}
<p class="empty">Nothing to give a quota to yet. Make an organization on the
<a href="/admin/users">people page</a> first.</p>
<a href="/admin/users">Organizations &amp; people</a> tab first.</p>
{{end}}
<h2 id="people-limits">People</h2>
@@ -198,7 +183,7 @@ asked. A model's owner is set on its card on the mirror page.</p>
{{if .CapForm.Scope}}<a href="/admin/quotas#capacity">Cancel</a>{{end}}</div>
</form>
{{else}}<p class="empty">Nothing to give a capacity quota to yet. Make an organization on the
<a href="/admin/users">people page</a> first.</p>{{end}}
<a href="/admin/users">Organizations &amp; people</a> tab first.</p>{{end}}
{{end}}
<details class="explain">
@@ -211,10 +196,8 @@ asked. A model's owner is set on its card on the mirror page.</p>
putting every caller's throughput one click from an admin session is more
than this page should do — <code>llmgw keys limits --scope key</code>.</p>
</details>
</div>
{{end}}
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
</div></body></html>{{end}}
{{/* A limit as the table shows it: the row's own value, else the one it is
held to from above, marked as inherited and saying from where. */}}
@@ -224,6 +207,8 @@ asked. A model's owner is set on its card on the mirror page.</p>
<input type="hidden" name="csrf" value="{{.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}">
<input type="hidden" name="id" value="{{.ID}}">
{{/* From a panel on the people tab: the answer comes back to that panel. */}}
{{with .Sel}}<input type="hidden" name="tab" value="people"><input type="hidden" name="sel" value="{{.}}">{{end}}
<p class="q-form-title"><strong>{{.Title}}</strong>. {{.Note}} A blank field means no limit of that kind.</p>
<div class="field-row">
<label>Requests / minute <input type="text" name="rpm" value="{{.F.RPM}}" placeholder="none"></label>
@@ -239,7 +224,7 @@ asked. A model's owner is set on its card on the mirror page.</p>
<input type="text" name="models" value="{{.F.Models}}" placeholder="every model"></label>
<div class="q-form-actions">
<button type="submit">Save</button>
<a href="/admin/quotas">Cancel</a>
{{if not .Sel}}<a href="/admin/quotas">Cancel</a>{{end}}
<button type="submit" name="clear" value="1" class="link danger"
{{if not .F.HasAny}}disabled{{end}}>Remove these limits</button>
</div>
+4 -13
View File
@@ -1,13 +1,5 @@
{{define "admin_usage"}}<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Usage · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>Usage over time</h1>
<p class="sub">Who is using the cluster, and when</p>
{{template "nav" .Nav}}
</header>
{{/* The usage tab of the people page. */}}
{{define "usage_body"}}
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
@@ -88,7 +80,6 @@ is every group, in the same order.</p>
<p class="note">A person is not a level of the quota tree — one person holds
keys in several projects — so grouping by person crosses the tree rather than
naming a level of it. Quotas are written on the other three, on the
<a href="/admin/quotas">quotas page</a>.</p>
<a href="/admin/quotas">Limits</a> tab.</p>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
</div></body></html>{{end}}
{{end}}
+10 -58
View File
@@ -1,21 +1,5 @@
{{define "admin_users"}}<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>People · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
{{/* Forms here post and get the whole page back, which works with no script at
all. htmx, when it loads, swaps only #people from that page instead of
reloading it, so an open project stays open and the page does not jump. */}}
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false}'>
<script src="/_static/htmx-2.0.11.min.js" defer></script>
</head><body><div class="wrap">
<header>
<h1>People</h1>
<p class="sub">{{len .Pending}} waiting for approval · {{len .Accounts}} accounts · {{len .Projects}} projects</p>
{{template "nav" .Nav}}
</header>
<div id="people" hx-boost="true" hx-target="#people" hx-select="#people"
hx-swap="outerHTML" hx-push-url="false">
{{/* The first tab of the people page: the tree, and everyone in it. */}}
{{define "people_body"}}
{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
{{if .Message}}<p class="ok" role="status">{{.Message}}</p>{{end}}
@@ -47,8 +31,9 @@
<h2>Organizations, teams and projects</h2>
<p class="note">A person is in a project, a project in a team, a team in an
organization{{if .HasQuotas}}, and <a href="/admin/quotas">quotas</a> can be set
on any of them{{end}}. Open a project to see its people and keys.</p>
organization{{if .HasQuotas}}, and <a href="/admin/quotas">limits</a> can be set
on any of them{{end}}. Choose a name — or a person in the list below — to see
its limits, capacity and use beside the tree; open a project for its people and keys.</p>
{{if not .Tree}}
<p class="empty">No organizations yet. Create one, then a team in it, then a
@@ -58,7 +43,7 @@ project in the team — an account can only be approved into a project.</p>
{{range $org := .Tree}}
<section class="org">
<div class="node-head">
<strong class="node-name">{{$org.Name}}</strong>
<strong class="node-name"><a href="/admin/users?sel=org:{{$org.ID}}" class="sel-link{{if eq $.Sel (printf "org:%s" $org.ID)}} on{{end}}">{{$org.Name}}</a></strong>
<span class="node-meta">{{len $org.Teams}} team{{if ne (len $org.Teams) 1}}s{{end}}</span>
{{template "scope-members" (dict "Kind" "org" "ID" $org.ID "Name" $org.Name "D" $)}}
<details class="edit"><summary>rename</summary>
@@ -80,7 +65,7 @@ project in the team — an account can only be approved into a project.</p>
{{range $team := $org.Teams}}
<div class="team">
<div class="node-head">
<span class="node-name">{{$team.Name}}</span>
<span class="node-name"><a href="/admin/users?sel=team:{{$team.ID}}" class="sel-link{{if eq $.Sel (printf "team:%s" $team.ID)}} on{{end}}">{{$team.Name}}</a></span>
<span class="node-meta">{{len $team.Projects}} project{{if ne (len $team.Projects) 1}}s{{end}}</span>
{{template "scope-members" (dict "Kind" "team" "ID" $team.ID "Name" $team.Name "D" $)}}
<details class="edit"><summary>rename</summary>
@@ -114,6 +99,7 @@ project in the team — an account can only be approved into a project.</p>
<details class="project" id="p-{{$p.ID}}"{{if eq $.Open $p.ID}} open{{end}}>
<summary>
<span class="node-name">{{$p.Name}}</span>
<a href="/admin/users?sel=project:{{$p.ID}}&amp;open={{$p.ID}}" class="sel-link small{{if eq $.Sel (printf "project:%s" $p.ID)}} on{{end}}" title="Limits and use of {{$p.Name}}">limits &amp; use</a>
<span class="node-meta">{{$p.Users}} member{{if ne $p.Users 1}}s{{end}}
· {{$p.ActiveKeys}} key{{if ne $p.ActiveKeys 1}}s{{end}}{{if gt $p.Keys $p.ActiveKeys}} ({{sub $p.Keys $p.ActiveKeys}} revoked){{end}}</span>
</summary>
@@ -222,42 +208,8 @@ and issuing a key for someone else. Each turns a stolen admin session into more
access than it had, so they need a shell on the control plane —
<code>llmgw users promote</code>, <code>llmgw users reset-password</code>,
<code>llmgw keys create</code>.</p>
</div>
{{end}}
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
</div>
<script>
// Search the account list. Outside #people so what was typed survives a swap,
// and applied again after every swap.
(function () {
var box = document.createElement('input');
box.type = 'search'; box.placeholder = 'Find a person by email, name or project';
box.className = 'search'; box.setAttribute('aria-label', 'Find a person');
function apply() {
var q = box.value.trim().toLowerCase();
document.querySelectorAll('#account-list tr[data-q]').forEach(function (tr) {
tr.hidden = q !== '' && tr.dataset.q.indexOf(q) < 0;
});
}
box.addEventListener('input', apply);
function place() {
var h = document.getElementById('accounts');
if (h && box.parentNode !== h.parentNode) h.after(box);
apply();
}
place();
// Projects the reader had open stay open across a change.
var open = [];
document.addEventListener('htmx:beforeSwap', function () {
open = Array.prototype.map.call(document.querySelectorAll('details.project[open]'), function (d) { return d.id; });
});
document.addEventListener('htmx:afterSettle', function () {
open.forEach(function (id) { var d = document.getElementById(id); if (d) d.open = true; });
place();
});
})();
</script>
</body></html>{{end}}
{{/* The actions every account row carries: suspend or reinstate, remove. */}}
{{define "state-actions"}}{{$u := .U}}{{$d := .D}}
@@ -295,7 +247,7 @@ access than it had, so they need a shell on the control plane —
they could be. */}}
{{define "account-row"}}{{$u := .U}}{{$d := .D}}
<tr data-q="{{lower $u.Email}} {{lower $u.Name}}{{range $u.Memberships}} {{lower .Path}}{{end}}">
<td>{{$u.Email}}{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
<td><a href="/admin/users?sel=user:{{$u.ID}}" class="sel-link{{if eq $d.Sel (printf "user:%s" $u.ID)}} on{{end}}">{{$u.Email}}</a>{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
<td class="state state-{{$u.State}}">{{$u.State}}</td>
<td class="where">
{{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
+6 -3
View File
@@ -29,7 +29,7 @@
{{if .Models}}
<div class="chat-bar">
<label class="inline-label">Model
<select id="model">{{range .Models}}<option value="{{.Name}}" data-simulated="{{if .Simulated}}1{{else}}0{{end}}" data-ondemand="{{if .OnDemand}}1{{else}}0{{end}}">{{.Name}}{{if .Simulated}} (simulated){{end}}{{if .OnDemand}} (not loaded){{end}}</option>{{end}}</select>
<select id="model">{{range .Models}}<option value="{{.Name}}" data-simulated="{{if .Simulated}}1{{else}}0{{end}}" data-ondemand="{{if .OnDemand}}1{{else}}0{{end}}">{{.Name}}{{with .Runs}} · {{.}}{{end}}{{if .Simulated}} (simulated){{end}}{{if .OnDemand}} (not loaded){{end}}</option>{{end}}</select>
</label>
<button id="reset" type="button" class="link">New conversation</button>
</div>
@@ -301,10 +301,13 @@ access to it.</p>
let busy = null;
// A line under a reply: how long it took and how fast it came.
function stats(out, tokens, t0, tFirst, tEnd, stopped) {
function stats(out, tokens, t0, tFirst, tEnd, stopped, hw) {
const p = document.createElement("p");
p.className = "note chat-stats";
const parts = [];
// What answered: a model with replicas on a card and on CPUs is served
// by either, and the speed below means nothing without knowing which.
if (hw) parts.push(hw === "cpu" ? "CPU" : "GPU " + hw);
if (tokens) parts.push(tokens + " tokens");
if (tokens && tFirst && tEnd > tFirst) parts.push((tokens / ((tEnd - tFirst) / 1000)).toFixed(1) + " tok/s");
if (tFirst) parts.push("first token " + ((tFirst - t0) / 1000).toFixed(2) + " s");
@@ -428,7 +431,7 @@ access to it.</p>
note.textContent = "Cut off at the length limit — ask it to continue.";
out.after(note);
}
stats(out, tokens || pieces, t0, tFirst, performance.now(), stopped);
stats(out, tokens || pieces, t0, tFirst, performance.now(), stopped, resp.headers.get("x-llmgw-hardware"));
// Once, at the end. Re-rendering every chunk would try to typeset half
// an expression on each token and flicker its way through the reply.
await renderReply(out, reply);
+3 -7
View File
@@ -8,17 +8,13 @@
{{- end}}
{{- if .Admin}}
<a href="/admin"{{if eq .Current "cluster"}} aria-current="page"{{end}}>Cluster</a>
{{- /* Limits and usage are tabs of the people page, not pages of their own. */}}
<a href="/admin/users"{{if eq .Current "people"}} aria-current="page"{{end}}>People</a>
{{- if .Quotas}}
<a href="/admin/quotas"{{if eq .Current "quotas"}} aria-current="page"{{end}}>Quotas</a>
<a href="/admin/usage"{{if eq .Current "usage"}} aria-current="page"{{end}}>Usage</a>
{{- end}}
{{- if .Mirror}}
<a href="/admin/mirror"{{if eq .Current "mirror"}} aria-current="page"{{end}}>Mirror</a>
{{- end}}
{{- if .Org}}
<a href="/org"{{if eq .Current "org"}} aria-current="page"{{end}}>Organizations</a>
{{- end}}
{{- /* An organization is a panel of the people page for an administrator;
/org is the managers' view, and leads an administrator there. */}}
<span class="menu-gap" aria-hidden="true"></span>
{{- end}}
<a href="/chat"{{if eq .Current "chat"}} aria-current="page"{{end}}>Chat</a>