The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

Every template and handler message in internal/web goes through a JSON
catalogue under internal/web/locales/<code>/, with English as the fallback.
The language is the person's choice (a cookie set by /_lang/{code}), else
Accept-Language, else landing.language. Tests refuse a key some language
lacks, a translation with other arguments than English, and template text
left in English.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:12:53 +02:00
1 parent 30f7f3474a
commit 11231d8c2b
113 files changed
+8091 -1981

No files matched your search

+123 -162
View File
@@ -1,7 +1,7 @@
{{define "about"}}{{if not .Site}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="description" content="llmgw serves models on your own GPUs behind one OpenAI-compatible API.">
<meta name="description" content="{{t "about.meta.description"}}">
{{end}}<title>llmgw</title>
{{if .Site}}<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
@@ -176,52 +176,50 @@ footer a{color:var(--muted)}
<div class="wrap">
<header class="bar">
<a class="mark" href="#top" aria-label="llmgw home">
<a class="mark" href="#top" aria-label="{{t "about.home_label"}}">
<svg width="26" height="26" viewBox="0 0 26 26" aria-hidden="true">
<rect x="1" y="1" width="24" height="24" rx="6" fill="none" stroke="currentColor" stroke-width="2"/>
<path d="M7 9h12M7 13h12M7 17h12" stroke="var(--fiber)" stroke-width="2" stroke-linecap="round"/>
</svg>
llmgw
</a>
<nav aria-label="Sections">
<a href="#what">What it does</a>
<a href="#fabrics">Fabrics</a>
<a href="#install">Install</a>
<a href="#proxy">Reverse proxy</a>
<a href="#download">Download</a>
<a href="https://gitea.mynodes.xyz/ilker/llmgw">Source</a>
{{if not .Site}}<a href="/">This gateway</a>{{end}}
<nav aria-label="{{t "about.nav.label"}}">
<a href="#what">{{t "about.nav.what"}}</a>
<a href="#fabrics">{{t "about.nav.fabrics"}}</a>
<a href="#install">{{t "about.nav.install"}}</a>
<a href="#proxy">{{t "about.nav.proxy"}}</a>
<a href="#download">{{t "about.nav.download"}}</a>
<a href="https://gitea.mynodes.xyz/ilker/llmgw">{{t "about.nav.source"}}</a>
{{if not .Site}}<a href="/">{{t "about.nav.gateway"}}</a>{{end}}
</nav>
</header>
<div class="hero" id="top">
<div>
<p class="eyebrow">Self-hosted LLM gateway · {{.Version}}</p>
<h1 style="margin-top:14px">One endpoint in front of <em>your own GPUs</em>.</h1>
<p class="lede">llmgw runs vLLM, SGLang and llama.cpp across your servers and serves them
behind one OpenAI-compatible API. It knows which GPUs share an NVLink domain or an InfiniBand
fabric, and a model split across machines never straddles two.</p>
<p class="eyebrow">{{t "about.hero.eyebrow" .Version}}</p>
<h1 style="margin-top:14px">{{th "about.hero.title"}}</h1>
<p class="lede">{{t "about.hero.lede"}}</p>
<div class="cta">
<a class="btn" href="#download">Download{{with .Release}} {{.}}{{end}}</a>
<a class="btn ghost" href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/quick-install.md">Install guide</a>
<a class="btn" href="#download">{{t "about.hero.download"}}{{with .Release}} {{.}}{{end}}</a>
<a class="btn ghost" href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/quick-install.md">{{t "about.hero.guide"}}</a>
</div>
<div class="oneline" aria-label="Set up a control plane">
<div class="oneline" aria-label="{{t "about.hero.oneline_label"}}">
<code><span class="p">$ </span>sudo llmgw server init --admin you@example.com</code>
<button class="copy" style="position:static" data-copy="sudo llmgw server init --admin you@example.com" type="button">Copy</button>
<button class="copy" style="position:static" data-copy="sudo llmgw server init --admin you@example.com" type="button">{{t "about.copy"}}</button>
</div>
</div>
<figure class="schem">
<svg viewBox="0 0 440 250" role="img" aria-labelledby="schem-t">
<title id="schem-t">Clients reach one llmgw address. llmgw places a model's shards on GPUs inside one fabric at a time.</title>
<title id="schem-t">{{t "about.schem.title"}}</title>
<!-- clients -->
<g>
<rect class="s-box" x="4" y="58" width="92" height="30" rx="6"/>
<text class="s-label" x="50" y="77" text-anchor="middle">openai sdk</text>
<text class="s-label" x="50" y="77" text-anchor="middle" translate="no">openai sdk</text>
<rect class="s-box" x="4" y="110" width="92" height="30" rx="6"/>
<text class="s-label" x="50" y="129" text-anchor="middle">editor agent</text>
<text class="s-label" x="50" y="129" text-anchor="middle">{{t "about.schem.editor"}}</text>
<rect class="s-box" x="4" y="162" width="92" height="30" rx="6"/>
<text class="s-label" x="50" y="181" text-anchor="middle">curl</text>
<text class="s-label" x="50" y="181" text-anchor="middle" translate="no">curl</text>
</g>
<path id="c1" class="s-link" d="M96 73 C130 73 130 118 160 118"/>
<path id="c2" class="s-link" d="M96 125 H160"/>
@@ -234,7 +232,7 @@ footer a{color:var(--muted)}
<path id="fb" class="s-link" d="M244 136 C272 136 272 188 300 188"/>
<!-- fabric a -->
<rect class="s-fab" x="300" y="14" width="136" height="98" rx="10"/>
<text class="s-label" x="312" y="32">nvl72 · gb200</text>
<text class="s-label" x="312" y="32" translate="no">nvl72 · gb200</text>
<g transform="translate(312 42)">
<rect class="s-gpu-a" width="22" height="22" rx="3"/><rect class="s-gpu-a" x="28" width="22" height="22" rx="3"/>
<rect class="s-gpu-a" x="56" width="22" height="22" rx="3"/><rect class="s-gpu-a" x="84" width="22" height="22" rx="3"/>
@@ -243,7 +241,7 @@ footer a{color:var(--muted)}
</g>
<!-- fabric b -->
<rect class="s-fab" x="300" y="138" width="136" height="98" rx="10"/>
<text class="s-label" x="312" y="156">ib-b · h100-80g</text>
<text class="s-label" x="312" y="156" translate="no">ib-b · h100-80g</text>
<g transform="translate(312 166)">
<rect class="s-gpu-b" width="22" height="22" rx="3"/><rect class="s-gpu-b" x="28" width="22" height="22" rx="3"/>
<rect class="s-gpu-idle" x="56" width="22" height="22" rx="3"/><rect class="s-gpu-idle" x="84" width="22" height="22" rx="3"/>
@@ -257,50 +255,37 @@ footer a{color:var(--muted)}
<circle class="pulse" r="4"><animateMotion dur="2.4s" begin="1.2s" repeatCount="indefinite"><mpath href="#fa"/></animateMotion></circle>
<circle class="pulse b" r="4"><animateMotion dur="2.4s" begin="2s" repeatCount="indefinite"><mpath href="#fb"/></animateMotion></circle>
</svg>
<figcaption>A replica spanning nodes goes into one NVLink domain if one has room, otherwise onto
one InfiniBand fabric. Two fabrics that cannot see each other are never bridged: the all-reduce
would hang.</figcaption>
<figcaption>{{t "about.schem.caption"}}</figcaption>
</figure>
</div>
<section id="what">
<div class="sec-head">
<div>
<h2>What it does</h2>
<p class="intro">One static binary. It runs as the control plane, as the agent on every GPU
node, and as the weights mirror.</p>
<h2>{{t "about.what.title"}}</h2>
<p class="intro">{{t "about.what.intro"}}</p>
</div>
<div class="facts">
<div class="fact"><h3>OpenAI API, your hardware</h3>
<p>Point any OpenAI client at it. Only the base URL changes. Streaming, tool calls,
<code>/v1/models</code> and <code>/v1/responses</code> work as your client expects.</p></div>
<div class="fact"><h3>Placement that knows the fabric</h3>
<p>Agents report GPU class, memory, NVLink peers and InfiniBand ports. Replicas go where they
fit, including ones larger than one machine, on the GPUs with the most NVLink pairs.</p></div>
<div class="fact"><h3>Models load when asked for</h3>
<p>Keep more models in the catalog than your GPUs hold. The first request starts one, the
chat page shows it loading, and it stops again after the idle timeout.</p></div>
<div class="fact"><h3>Keys, budgets, rate limits</h3>
<p>Organisations, teams and projects, each with limits. Priority bands decide who waits
when the cluster is full. Managers see their organisation's spend; people see their own.</p></div>
<div class="fact"><h3>Editors connect from the account page</h3>
<p>Pick Continue, Codex or Cline and a model. The page makes a key and hands over the config,
filled in, or a one-line command that writes it.</p></div>
<div class="fact"><h3>A chat page for quick questions</h3>
<p>Markdown and maths rendered, stop mid-reply, tokens per second and time to first token,
a system prompt and a temperature. The gateway keeps no conversations.</p></div>
<div class="fact"><h3>A mirror you browse</h3>
<p>Explore what is trending on Hugging Face, read a model card under its row, and download
once for the whole cluster. Only what your nodes can run is shown by default.</p></div>
<div class="fact"><h3>Nodes install their own engines</h3>
<p>llama.cpp on every node; vLLM on every GPU node that can run it, from PyPI with every
file hash-checked. A node that cannot run it says why.</p></div>
<div class="fact"><h3>Upgrades that keep models running</h3>
<p>Upgrade the control plane, then the agents. A restarted agent adopts the engines it was
running instead of reloading them, so replicas stay up throughout.</p></div>
<div class="fact"><h3>A dashboard with nothing to set up</h3>
<p><code>/admin</code> shows capacity per GPU class, CPU and memory per node, who is using
what, each node's page and logs. It fits a phone. Prometheus is optional.</p></div>
<div class="fact"><h3>{{t "about.what.api.title"}}</h3>
<p>{{th "about.what.api.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.placement.title"}}</h3>
<p>{{t "about.what.placement.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.ondemand.title"}}</h3>
<p>{{t "about.what.ondemand.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.keys.title"}}</h3>
<p>{{t "about.what.keys.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.editors.title"}}</h3>
<p>{{t "about.what.editors.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.chat.title"}}</h3>
<p>{{t "about.what.chat.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.mirror.title"}}</h3>
<p>{{t "about.what.mirror.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.engines.title"}}</h3>
<p>{{t "about.what.engines.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.upgrades.title"}}</h3>
<p>{{t "about.what.upgrades.text"}}</p></div>
<div class="fact"><h3>{{t "about.what.dashboard.title"}}</h3>
<p>{{th "about.what.dashboard.text"}}</p></div>
</div>
</div>
</section>
@@ -308,24 +293,19 @@ footer a{color:var(--muted)}
<section id="fabrics">
<div class="sec-head">
<div>
<h2>InfiniBand and NVLink, several of each</h2>
<p class="intro">A site may have independent InfiniBand fabrics, eight-GPU HGX boards and NVL72
racks side by side. llmgw tells them apart by what the adapters report, not by what a config
file calls them.</p>
<p class="intro"><a href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/fabrics.md">Fabrics and NVLink guide</a></p>
<h2>{{t "about.fabrics.title"}}</h2>
<p class="intro">{{t "about.fabrics.intro"}}</p>
<p class="intro"><a href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/fabrics.md">{{t "about.fabrics.guide"}}</a></p>
</div>
<div class="fab-grid">
<div class="order" aria-label="Where a replica spanning nodes is placed, in order">
<div><b>1</b><h3>One NVLink domain</h3><p>NVL72 or NVLink Switch. Every card joined, IMEX up.
Shards get <code>NCCL_MNNVL_ENABLE=1</code>.</p></div>
<div><b>2</b><h3>One InfiniBand or RoCE fabric</h3><p>Nodes under one leaf first, nodes with a
degraded cable last.</p></div>
<div><b class="x">×</b><h3>Never across two</h3><p>Two fabrics given one name are placed as
two, and the cluster page says so.</p></div>
<div class="order" aria-label="{{t "about.fabrics.order_label"}}">
<div><b>1</b><h3>{{t "about.fabrics.nvlink.title"}}</h3><p>{{th "about.fabrics.nvlink.text"}}</p></div>
<div><b>2</b><h3>{{t "about.fabrics.ib.title"}}</h3><p>{{t "about.fabrics.ib.text"}}</p></div>
<div><b class="x">×</b><h3>{{t "about.fabrics.never.title"}}</h3><p>{{t "about.fabrics.never.text"}}</p></div>
</div>
<div>
<h3>Check a node in one command</h3>
<div class="term" aria-label="Example output of llmgw doctor"><pre>$ sudo llmgw doctor
<h3>{{t "about.fabrics.doctor"}}</h3>
<div class="term" aria-label="{{t "about.fabrics.doctor_label"}}"><pre>$ sudo llmgw doctor
<span class="fail">FAIL</span> nvlink this machine has NVSwitch but nvidia-fabricmanager is inactive
<span class="dim">→ systemctl enable --now nvidia-fabricmanager</span>
<span class="warn">WARN</span> fabric nvidia_peermem is not loaded: GPUDirect RDMA is off
@@ -337,14 +317,10 @@ footer a{color:var(--muted)}
<span class="ok">OK</span> agent NCCL bootstraps over ens3 (10.1.0.21)</pre></div>
</div>
<ul class="watch">
<li><strong>Subnet managers, watched</strong>Master, priority, standbys and failovers per fabric.
A fabric with no standby is flagged. llmgw does not run one.</li>
<li><strong>A map of each fabric</strong><code>/admin/fabrics</code> draws switches in tiers,
cables coloured by rate, degraded ones in red, and live traffic on your adapters.</li>
<li><strong>NVLink domains as racks</strong><code>/admin/nvlink</code> shows which trays are
joined, whose IMEX is down, which link runs degraded.</li>
<li><strong>Ports re-read every heartbeat</strong>A port that goes down takes its fabric off the
node within seconds. Counters and rates are exported to Prometheus.</li>
<li><strong>{{t "about.fabrics.sm.title"}}</strong>{{t "about.fabrics.sm.text"}}</li>
<li><strong>{{t "about.fabrics.map.title"}}</strong>{{th "about.fabrics.map.text"}}</li>
<li><strong>{{t "about.fabrics.racks.title"}}</strong>{{th "about.fabrics.racks.text"}}</li>
<li><strong>{{t "about.fabrics.ports.title"}}</strong>{{t "about.fabrics.ports.text"}}</li>
</ul>
</div>
</div>
@@ -353,44 +329,38 @@ footer a{color:var(--muted)}
<section id="install">
<div class="sec-head">
<div>
<h2>From nothing to a first request</h2>
<p class="intro">Debian, Ubuntu or RHEL 10. NVIDIA drivers and the InfiniBand stack are
yours to install; llmgw does the rest.</p>
<p class="intro"><a href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/install.md">The full guide</a>
covers the mirror, GPUs and what to do when something does not work.</p>
<h2>{{t "about.install.title"}}</h2>
<p class="intro">{{t "about.install.intro"}}</p>
<p class="intro">{{th "about.install.guide"}}</p>
</div>
<ol class="steps">
<li>
<h3>Put the binary on every machine</h3>
<p>Static, no dependencies, the same file everywhere.</p>
<div class="code"><button class="copy" type="button">Copy</button><pre><code>VERSION={{or .Release "vX.Y.Z"}}{{if not .Release}} <span class="c"># from the releases page</span>{{end}}
<h3>{{t "about.install.binary.title"}}</h3>
<p>{{t "about.install.binary.text"}}</p>
<div class="code"><button class="copy" type="button">{{t "about.copy"}}</button><pre><code>VERSION={{or .Release "vX.Y.Z"}}{{if not .Release}} <span class="c"># from the releases page</span>{{end}}
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw</code></pre></div>
</li>
<li>
<h3>Set up the control plane</h3>
<p>Creates the user, the certificate authority, the certificate, <code>server.yaml</code>
and your administrator account, then starts the service. Safe to run again.
Add <code>--host</code> for any name nodes use that the machine cannot see itself.</p>
<div class="code"><button class="copy" type="button">Copy</button><pre><code>sudo llmgw server init --admin you@example.com</code></pre></div>
<h3>{{t "about.install.server.title"}}</h3>
<p>{{th "about.install.server.text"}}</p>
<div class="code"><button class="copy" type="button">{{t "about.copy"}}</button><pre><code>sudo llmgw server init --admin you@example.com</code></pre></div>
</li>
<li>
<h3>Add a node</h3>
<p>From the control plane, which holds the authority. The node gets its certificate,
the binary and the agent, and installs its engine by itself.</p>
<div class="code"><button class="copy" type="button">Copy</button><pre><code>deploy/bootstrap/onboard-node.sh --node gpu01 --addr 10.0.0.141 \
<h3>{{t "about.install.node.title"}}</h3>
<p>{{t "about.install.node.text"}}</p>
<div class="code"><button class="copy" type="button">{{t "about.copy"}}</button><pre><code>deploy/bootstrap/onboard-node.sh --node gpu01 --addr 10.0.0.141 \
--server https://10.0.0.140:7443 --fabric ib-a:mlx5_0,mlx5_1</code></pre></div>
<div class="term" aria-label="Example output of llmgw ctl nodes"><pre>$ llmgw ctl nodes
<div class="term" aria-label="{{t "about.install.node.output_label"}}"><pre>$ llmgw ctl nodes
NODE STATE GPUS MEM CLASSES FABRICS REPLICAS LAST SEEN
gpu01 <span class="ok">ready</span> 8/8 free 503G h100-80g ib-a 0 2s</pre></div>
</li>
<li>
<h3>Add a model and send a request</h3>
<p>No restart. If it does not place, <code>llmgw ctl explain-placement</code> names every
node and why it was turned down.</p>
<div class="code"><button class="copy" type="button">Copy</button><pre><code>llmgw ctl model add -f qwen-coder-3b.yaml
<h3>{{t "about.install.model.title"}}</h3>
<p>{{th "about.install.model.text"}}</p>
<div class="code"><button class="copy" type="button">{{t "about.copy"}}</button><pre><code>llmgw ctl model add -f qwen-coder-3b.yaml
curl https://llm.example.com/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
@@ -404,35 +374,32 @@ curl https://llm.example.com/v1/chat/completions \
<section id="proxy">
<div class="sec-head">
<div>
<h2>Behind a reverse proxy</h2>
<p class="intro">Port 8080 is plain HTTP. Put it behind Caddy or nginx with a public
certificate. Only 8080 goes through the proxy: agents reach 7443 directly, because it is
mutual TLS with the cluster's own authority.</p>
<p class="intro">Bind llmgw to loopback when the proxy is on the same machine:
<code>listen: "127.0.0.1:8080"</code>.</p>
<h2>{{t "about.proxy.title"}}</h2>
<p class="intro">{{t "about.proxy.intro"}}</p>
<p class="intro">{{th "about.proxy.loopback"}}</p>
</div>
<div class="tabs">
<input type="radio" name="proxy" id="t-caddy" checked>
<input type="radio" name="proxy" id="t-nginx">
<div class="tablist" role="presentation">
<div class="tablist" role="presentation" translate="no">
<label for="t-caddy">Caddy</label>
<label for="t-nginx">nginx</label>
</div>
<div class="panel p-caddy">
<p class="meta"><span>Obtains and renews its own certificate. Replace <code>llm.example.com</code>.</span>
<a href="{{.Download}}/Caddyfile">Caddyfile</a></p>
<div class="code"><button class="copy" type="button">Copy</button><pre><code>{{.Caddy}}</code></pre></div>
<p class="meta"><span>{{th "about.proxy.caddy"}}</span>
<a href="{{.Download}}/Caddyfile" translate="no">Caddyfile</a></p>
<div class="code"><button class="copy" type="button">{{t "about.copy"}}</button><pre><code>{{.Caddy}}</code></pre></div>
</div>
<div class="panel p-nginx">
<p class="meta"><span>Certificate from <code>certbot --nginx</code>. Works on nginx 1.18 and later.</span>
<a href="{{.Download}}/nginx.conf">nginx.conf</a></p>
<div class="code"><button class="copy" type="button">Copy</button><pre><code>{{.Nginx}}</code></pre></div>
<p class="meta"><span>{{th "about.proxy.nginx"}}</span>
<a href="{{.Download}}/nginx.conf" translate="no">nginx.conf</a></p>
<div class="code"><button class="copy" type="button">{{t "about.copy"}}</button><pre><code>{{.Nginx}}</code></pre></div>
</div>
<ul class="rules">
<li><strong>Streams are not buffered</strong>Tokens reach the client as they are generated, not in one burst at the end.</li>
<li><strong>The read timeout outlasts the model</strong>nginx's 60-second default turns a slow first token into a 504.</li>
<li><strong>Forwarded headers are set, not passed on</strong>llmgw marks session cookies Secure from <code>X-Forwarded-Proto</code> and shows <code>X-Forwarded-Host</code> as the base URL.</li>
<li><strong>Body limit matches the router</strong>32 MiB, so an oversized request is refused before it is uploaded.</li>
<li><strong>{{t "about.proxy.stream.title"}}</strong>{{t "about.proxy.stream.text"}}</li>
<li><strong>{{t "about.proxy.timeout.title"}}</strong>{{t "about.proxy.timeout.text"}}</li>
<li><strong>{{t "about.proxy.headers.title"}}</strong>{{th "about.proxy.headers.text"}}</li>
<li><strong>{{t "about.proxy.body.title"}}</strong>{{t "about.proxy.body.text"}}</li>
</ul>
</div>
</div>
@@ -441,33 +408,29 @@ curl https://llm.example.com/v1/chat/completions \
<section id="download">
<div class="sec-head">
<div>
<h2>Download</h2>
<p class="intro">Built by CI from the tag and checked against <code>checksums.txt</code>.
Every release is on the <a href="https://gitea.mynodes.xyz/ilker/llmgw/releases">releases page</a>
with its changelog.</p>
<h2>{{t "about.download.title"}}</h2>
<p class="intro">{{th "about.download.intro"}}</p>
</div>
<div>
{{if .Release}}
<div class="ver"><span class="tag">{{.Release}}</span>{{with .Date}}<span class="eyebrow">Released {{.}}</span>{{end}}</div>
<div class="ver"><span class="tag">{{.Release}}</span>{{with .Date}}<span class="eyebrow">{{t "about.download.released" .}}</span>{{end}}</div>
<div class="dl-wrap">
<table class="dl">
<thead><tr><th>For</th><th>File</th><th>Notes</th></tr></thead>
<thead><tr><th>{{t "about.download.col.for"}}</th><th>{{t "about.download.col.file"}}</th><th>{{t "about.download.col.notes"}}</th></tr></thead>
<tbody>
<tr><td>Linux x86-64</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-linux-amd64"><code>llmgw-{{$.Release}}-linux-amd64</code></a></td><td class="note">Static. Runs on any libc.</td></tr>
<tr><td>Linux ARM64</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-linux-arm64"><code>llmgw-{{$.Release}}-linux-arm64</code></a></td><td class="note">Static.</td></tr>
<tr><td>Linux x86-64, NVLink</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-linux-amd64-nvml"><code>llmgw-{{$.Release}}-linux-amd64-nvml</code></a></td><td class="note">Sees NVLink topology through NVML. For PCIe boxes with NVLink bridges. Needs the driver.</td></tr>
<tr><td>macOS Apple silicon</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-darwin-arm64"><code>llmgw-{{$.Release}}-darwin-arm64</code></a></td><td class="note">For <code>llmgw ctl</code> and trying it on a laptop.</td></tr>
<tr><td>macOS Intel</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-darwin-amd64"><code>llmgw-{{$.Release}}-darwin-amd64</code></a></td><td class="note"></td></tr>
<tr><td>Caddy</td><td><a href="{{$.Download}}/Caddyfile"><code>Caddyfile</code></a></td><td class="note">Reverse proxy for port 8080.</td></tr>
<tr><td>nginx</td><td><a href="{{$.Download}}/nginx.conf"><code>nginx.conf</code></a></td><td class="note">Reverse proxy for port 8080.</td></tr>
<tr><td>Checksums</td><td><a href="{{$.Download}}/checksums.txt"><code>checksums.txt</code></a></td><td class="note">SHA-256 of every file above.</td></tr>
<tr><td translate="no">Linux x86-64</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-linux-amd64"><code>llmgw-{{$.Release}}-linux-amd64</code></a></td><td class="note">{{t "about.download.static_any"}}</td></tr>
<tr><td translate="no">Linux ARM64</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-linux-arm64"><code>llmgw-{{$.Release}}-linux-arm64</code></a></td><td class="note">{{t "about.download.static"}}</td></tr>
<tr><td translate="no">Linux x86-64, NVLink</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-linux-amd64-nvml"><code>llmgw-{{$.Release}}-linux-amd64-nvml</code></a></td><td class="note">{{t "about.download.nvml"}}</td></tr>
<tr><td>{{t "about.download.mac_arm"}}</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-darwin-arm64"><code>llmgw-{{$.Release}}-darwin-arm64</code></a></td><td class="note">{{th "about.download.mac_note"}}</td></tr>
<tr><td translate="no">macOS Intel</td><td><a href="{{$.Download}}/llmgw-{{$.Release}}-darwin-amd64"><code>llmgw-{{$.Release}}-darwin-amd64</code></a></td><td class="note"></td></tr>
<tr><td translate="no">Caddy</td><td><a href="{{$.Download}}/Caddyfile"><code>Caddyfile</code></a></td><td class="note">{{t "about.download.proxy_note"}}</td></tr>
<tr><td translate="no">nginx</td><td><a href="{{$.Download}}/nginx.conf"><code>nginx.conf</code></a></td><td class="note">{{t "about.download.proxy_note"}}</td></tr>
<tr><td>{{t "about.download.checksums"}}</td><td><a href="{{$.Download}}/checksums.txt"><code>checksums.txt</code></a></td><td class="note">{{t "about.download.checksums_note"}}</td></tr>
</tbody>
</table>
</div>
{{else}}
<p class="intro" style="margin-top:0">This is a development build ({{.Version}}). Take a release
from the <a href="https://gitea.mynodes.xyz/ilker/llmgw/releases">releases page</a>, where every
file is listed with its checksum.</p>
<p class="intro" style="margin-top:0">{{th "about.download.dev" .Version}}</p>
{{end}}
</div>
</div>
@@ -476,30 +439,27 @@ curl https://llm.example.com/v1/chat/completions \
<section id="status">
<div class="sec-head">
<div>
<h2>Where it stands</h2>
<p class="intro">Pre-1.0: configuration formats may change between minor versions.
This is what has run on real hardware, and what has not yet.</p>
<h2>{{t "about.status.title"}}</h2>
<p class="intro">{{t "about.status.intro"}}</p>
</div>
<div class="status">
<div class="proven">
<h3>Run on real clusters</h3>
<h3>{{t "about.status.proven"}}</h3>
<ul>
<li>Single-node replicas on real GPUs and on CPU</li>
<li>On-demand loading and idle shutdown</li>
<li>Keys, budgets, rate limits and priority bands</li>
<li>Rolling upgrades with no dropped replicas</li>
<li>SQLite and PostgreSQL stores</li>
<li>Chat, editor setup and the Responses API against a six-node test cluster</li>
<li>{{t "about.status.proven.single"}}</li>
<li>{{t "about.status.proven.ondemand"}}</li>
<li>{{t "about.status.proven.keys"}}</li>
<li>{{t "about.status.proven.upgrades"}}</li>
<li>{{t "about.status.proven.stores"}}</li>
<li>{{t "about.status.proven.chat"}}</li>
</ul>
</div>
<div class="open">
<h3>Not yet proven</h3>
<h3>{{t "about.status.open"}}</h3>
<ul>
<li>InfiniBand and NVLink on a real site: written and tested against the real output of
<code>sminfo</code>, <code>ibnetdiscover</code> and <code>nvidia-smi</code>, but not yet run
on the hardware. Run <code>llmgw doctor --json</code> on a few nodes first.</li>
<li>Two control planes over long periods</li>
<li>Nodes do not share downloaded weights; the mirror keeps each download local</li>
<li>{{th "about.status.open.fabrics"}}</li>
<li>{{t "about.status.open.planes"}}</li>
<li>{{t "about.status.open.weights"}}</li>
</ul>
</div>
</div>
@@ -508,25 +468,26 @@ curl https://llm.example.com/v1/chat/completions \
<footer>
<span>llmgw {{.Version}}</span>
<span><a href="https://gitea.mynodes.xyz/ilker/llmgw">Source</a> ·
<a href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/CHANGELOG.md">Changelog</a> ·
<a href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/reference.md">Configuration reference</a></span>
<span><a href="https://gitea.mynodes.xyz/ilker/llmgw">{{t "about.nav.source"}}</a> ·
<a href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/CHANGELOG.md">{{t "about.footer.changelog"}}</a> ·
<a href="https://gitea.mynodes.xyz/ilker/llmgw/src/branch/main/docs/reference.md">{{t "about.footer.reference"}}</a></span>{{if not .Site}}{{template "langs"}}{{end}}
</footer>
</div>
<script>
var say = { copy: {{t "about.copy"}}, copied: {{t "about.copied"}}, selected: {{t "about.selected"}} };
document.querySelectorAll('.copy').forEach(function (b) {
b.addEventListener('click', function () {
var text = b.dataset.copy || b.parentElement.querySelector('pre').innerText;
var done = function (label) { b.textContent = label; setTimeout(function () { b.textContent = 'Copy'; }, 1600); };
var done = function (label) { b.textContent = label; setTimeout(function () { b.textContent = say.copy; }, 1600); };
if (navigator.clipboard && navigator.clipboard.writeText) {
navigator.clipboard.writeText(text).then(function () { done('Copied'); }, function () { select(); });
navigator.clipboard.writeText(text).then(function () { done(say.copied); }, function () { select(); });
} else { select(); }
function select() {
var el = b.dataset.copy ? b.parentElement.querySelector('code') : b.parentElement.querySelector('pre');
var r = document.createRange(); r.selectNodeContents(el);
var s = window.getSelection(); s.removeAllRanges(); s.addRange(r);
done('Selected');
done(say.selected);
}
});
});
+85 -95
View File
@@ -1,63 +1,59 @@
{{define "signup"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Sign up · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "signup.title"}}</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap narrow">
<header><h1>Create an account</h1>
<p class="sub">An operator approves it before it can be used.</p></header>
<header><h1>{{t "signup.heading"}}</h1>
<p class="sub">{{t "signup.sub"}}</p></header>
{{if .Done}}
<p>Thank you. Your account is waiting for approval.</p>
<p class="note">You will not get an email from this gateway — it does not send any.
Tell whoever you asked for access that you have signed up.</p>
<p><a href="/login">Sign in</a> once you have been approved.</p>
<p>{{t "signup.thanks"}}</p>
<p class="note">{{t "signup.no_email"}}</p>
<p>{{th "signup.sign_in_once_approved"}}</p>
{{/* The same page whether or not the address already had an account, so
it says what to do in either case without saying which this was. */}}
<p class="note">If this address already had an account, nothing new was made:
<a href="/login">sign in</a> with it, or, if you have forgotten its password, ask
an operator to reset it.</p>
<p class="note">{{th "signup.already_had"}}</p>
{{else}}
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
<form method="post" autocomplete="on">
<label>Email<input type="email" name="email" value="{{.Email}}" required autocomplete="email"></label>
<label>Name<input type="text" name="name" value="{{.Name}}" autocomplete="name"></label>
<label>Password<input type="password" name="password" required minlength="{{.MinPassword}}"
<label>{{t "signup.email"}}<input type="email" name="email" value="{{.Email}}" required autocomplete="email"></label>
<label>{{t "signup.name"}}<input type="text" name="name" value="{{.Name}}" autocomplete="name"></label>
<label>{{t "signup.password"}}<input type="password" name="password" required minlength="{{.MinPassword}}"
autocomplete="new-password"></label>
<p class="note">At least {{.MinPassword}} characters. Length is the only rule —
a long phrase beats a short one with punctuation in it.</p>
<button type="submit">Sign up</button>
<p class="note">{{t "signup.password_rule" .MinPassword}}</p>
<button type="submit">{{t "signup.submit"}}</button>
</form>
<p class="note">Already have an account? <a href="/login">Sign in</a>.</p>
<p class="note">{{th "signup.have_account"}}</p>
{{end}}
<footer><span>llmgw {{.Version}}</span><span><a href="/">about this gateway</a></span></footer>
<footer><span>llmgw {{.Version}}</span><span><a href="/">{{t "common.about_gateway"}}</a></span>{{template "langs"}}</footer>
</div></body></html>{{end}}
{{define "login"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Sign in · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "login.title"}}</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap narrow">
<header><h1>Sign in</h1></header>
<header><h1>{{t "login.heading"}}</h1></header>
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
<form method="post" autocomplete="on">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<label>Email<input type="email" name="email" value="{{.Email}}" required autocomplete="email"></label>
<label>Password<input type="password" name="password" required autocomplete="current-password"></label>
<button type="submit">Sign in</button>
<label>{{t "login.email"}}<input type="email" name="email" value="{{.Email}}" required autocomplete="email"></label>
<label>{{t "login.password"}}<input type="password" name="password" required autocomplete="current-password"></label>
<button type="submit">{{t "login.submit"}}</button>
</form>
{{if .SSO}}<p><a class="btn" href="/login/oidc">Sign in with {{.SSO}}</a></p>{{end}}
{{if .SignupOpen}}<p class="note">No account? <a href="/signup">Sign up</a>.</p>{{end}}
<footer><span>llmgw {{.Version}}</span><span><a href="/">about this gateway</a></span></footer>
{{if .SSO}}<p><a class="btn" href="/login/oidc">{{t "login.with_sso" .SSO}}</a></p>{{end}}
{{if .SignupOpen}}<p class="note">{{th "login.no_account"}}</p>{{end}}
<footer><span>llmgw {{.Version}}</span><span><a href="/">{{t "common.about_gateway"}}</a></span>{{template "langs"}}</footer>
</div></body></html>{{end}}
{{define "profile"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Account · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "profile.title"}}</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>Your account</h1>
<h1>{{t "profile.heading"}}</h1>
<p class="sub">{{.User.Email}}{{if ne .User.Name .User.Email}} ({{.User.Name}}){{end}}</p>
{{template "nav" .Nav}}
</header>
@@ -65,10 +61,10 @@ an operator to reset it.</p>
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
{{if .NewKey}}
<h2>Your new key</h2>
<p>This is shown once. Nothing can produce it again — only its hash is kept.</p>
<h2>{{t "profile.new_key"}}</h2>
<p>{{t "profile.shown_once"}}</p>
<pre>{{.NewKey}}</pre>
<p>Put it in a password manager now.</p>
<p>{{t "profile.store_it"}}</p>
<pre>curl {{.BaseURL}}/v1/chat/completions \
-H "Authorization: Bearer {{.NewKey}}" \
-H "Content-Type: application/json" \
@@ -77,57 +73,57 @@ an operator to reset it.</p>
{{with .Connect}}
<section class="connect-result" id="connected">
<h2>{{.ToolName}} is ready to connect</h2>
<p>A key was made for it, <strong>shown this once</strong> — it is in everything below. Nothing can produce it again.</p>
<h2>{{t "connect.ready" .ToolName}}</h2>
<p>{{th "connect.shown_once"}}</p>
{{if .Command}}
<h3>One command</h3>
<p class="note">Writes the file below{{if eq .Tool "codex"}} into your Codex configuration{{end}}, keeping a copy of anything it replaces.</p>
<div class="copyable"><pre>{{.Command}}</pre><button type="button" class="link copy">copy</button></div>
<h3>{{t "connect.one_command"}}</h3>
<p class="note">{{if eq .Tool "codex"}}{{t "connect.command_note_codex"}}{{else}}{{t "connect.command_note"}}{{end}}</p>
<div class="copyable"><pre>{{.Command}}</pre><button type="button" class="link copy">{{t "common.copy"}}</button></div>
{{end}}
{{range .Files}}
<h3>Or by hand: <code>{{.Path}}</code></h3>
<div class="copyable"><pre>{{.Body}}</pre><button type="button" class="link copy">copy</button>
<button type="button" class="link download-file" data-name="{{.Name}}">download</button></div>
<h3>{{th "connect.by_hand" .Path}}</h3>
<div class="copyable"><pre>{{.Body}}</pre><button type="button" class="link copy">{{t "common.copy"}}</button>
<button type="button" class="link download-file" data-name="{{.Name}}">{{t "common.download"}}</button></div>
{{end}}
{{if .Fields}}
<h3>Settings to fill in</h3>
<h3>{{t "connect.fields"}}</h3>
<table class="rows compact narrow-table connect-fields">{{range .Fields}}
<tr><th>{{.Label}}</th><td><code>{{.Value}}</code></td><td><button type="button" class="link copy" data-value="{{.Value}}">copy</button></td></tr>{{end}}
<tr><th>{{.Label}}</th><td><code>{{.Value}}</code></td><td><button type="button" class="link copy" data-value="{{.Value}}">{{t "common.copy"}}</button></td></tr>{{end}}
</table>
{{end}}
<h3>Then</h3>
<h3>{{t "connect.then"}}</h3>
<p>{{if eq .Tool "codex"}}<code>{{.Run}}</code>{{else}}{{.Run}}{{end}}</p>
{{range .Notes}}<p class="note">{{.}}</p>{{end}}
</section>
{{end}}
<h2>Connect your editor</h2>
<p class="note">A key of its own for the tool, and its configuration with the key, this gateway and a model
already in it. Revoke the key below to disconnect it.</p>
<h2>{{t "connect.heading"}}</h2>
<p class="note">{{t "connect.intro"}}</p>
<form method="post" action="/profile/connect#connected" class="connect-form">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<div class="field-row">
<label>Tool
<label>{{t "connect.tool"}}
<select name="tool">{{range .Tools}}<option value="{{.Key}}">{{.Name}}</option>{{end}}</select></label>
<label>Model
<label>{{t "connect.model"}}
{{if .Models}}<select name="model">{{range .Models}}<option value="{{.}}">{{.}}</option>{{end}}</select>
{{else}}<input type="text" name="model" placeholder="a model the gateway serves" required>{{end}}</label>
{{if .Contexts}}<label>Spends against
{{else}}<input type="text" name="model" placeholder="{{t "connect.model_placeholder"}}" required>{{end}}</label>
{{if .Contexts}}<label>{{t "connect.spends_against"}}
<select name="context">{{range .Contexts}}<option value="{{.Value}}">{{.Label}}</option>{{end}}</select></label>{{end}}
</div>
<button type="submit">Make a key and the configuration</button>
<button type="submit">{{t "connect.submit"}}</button>
</form>
<script>
// Copy and download for what the connect panel shows, done in the page: the
// key in it must not go back to the server to become a file.
const copiedWord = {{t "common.copied"}}, copyPrompt = {{t "profile.copy_prompt"}};
document.addEventListener("click", (e) => {
const b = e.target.closest("button.copy, button.download-file");
if (!b) return;
const text = b.dataset.value || b.parentElement.querySelector("pre").textContent;
if (b.classList.contains("copy")) {
const done = () => { const was = b.textContent; b.textContent = "copied"; setTimeout(() => (b.textContent = was), 1500); };
if (navigator.clipboard) navigator.clipboard.writeText(text).then(done, () => window.prompt("Copy", text));
else window.prompt("Copy", text);
const done = () => { const was = b.textContent; b.textContent = copiedWord; setTimeout(() => (b.textContent = was), 1500); };
if (navigator.clipboard) navigator.clipboard.writeText(text).then(done, () => window.prompt(copyPrompt, text));
else window.prompt(copyPrompt, text);
return;
}
const url = URL.createObjectURL(new Blob([text], { type: "text/plain" }));
@@ -138,88 +134,82 @@ document.addEventListener("click", (e) => {
});
</script>
<h2>Create a key</h2>
<h2>{{t "profile.create_key"}}</h2>
<form method="post" action="/profile/keys" class="narrow">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<label>What is it for?<input type="text" name="name" placeholder="laptop, ci, notebook" required></label>
<label>{{t "profile.key_for"}}<input type="text" name="name" placeholder="{{t "profile.key_for_placeholder"}}" required></label>
{{if .Contexts}}
<label>Spends against
<label>{{t "profile.spends_against"}}
<select name="context">
{{range .Contexts}}<option value="{{.Value}}">{{.Label}}</option>{{end}}
</select></label>
<button type="submit">Create</button>
<button type="submit">{{t "profile.create"}}</button>
{{else}}
<p class="note">You are in no project yet, and personal keys are off here.
Ask an administrator to add you to one.</p>
<p class="note">{{t "profile.no_project"}}</p>
{{end}}
</form>
<h2>Keys</h2>
<h2>{{t "profile.keys"}}</h2>
{{if .Keys}}
<table class="rows"><tr><th>Name</th><th>Key</th><th>Spends against</th><th>Created</th><th>Last used</th><th>State</th><th></th></tr>
<table class="rows"><tr><th>{{t "profile.col.name"}}</th><th>{{t "profile.col.key"}}</th><th>{{t "profile.col.spends_against"}}</th><th>{{t "profile.col.created"}}</th><th>{{t "profile.col.last_used"}}</th><th>{{t "profile.col.state"}}</th><th></th></tr>
{{range .Keys}}<tr>
<td>{{.Name}}</td>
<td><code>{{.Prefix}}</code></td>
<td>{{if .ProjectID}}{{or (index $.Where .ProjectID) "a project you have left"}}{{else}}Personal{{end}}</td>
<td>{{if .ProjectID}}{{or (index $.Where .ProjectID) (t "profile.left_project")}}{{else}}{{t "profile.personal"}}{{end}}</td>
<td>{{date .CreatedAt}}</td>
<td>{{ago .LastUsedAt}}</td>
<td class="state state-{{state .}}">{{state .}}</td>
<td class="state state-{{state .}}">{{t (print "profile.state." (state .))}}</td>
<td>{{if eq (state .) "active"}}
<form method="post" action="/profile/keys/{{.ID}}/revoke">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">Revoke</button>
<button type="submit" class="link danger">{{t "profile.revoke"}}</button>
</form>
{{end}}</td>
</tr>{{end}}</table>
<p class="note">Revoking takes effect within the gateway's cache window, up to a
minute. To rotate: create the new key, switch over, wait for "last used" on the
old one to stop moving, then revoke it.</p>
{{else}}<p class="empty">You have no keys yet.</p>{{end}}
<p class="note">{{t "profile.revoke_note"}}</p>
{{else}}<p class="empty">{{t "profile.no_keys"}}</p>{{end}}
<h2>Where you are</h2>
<h2>{{t "profile.where"}}</h2>
{{if .Memberships}}
<p>{{range $i, $m := .Memberships}}{{if $i}} · {{end}}{{$m.Path}} <span class="opt">{{$m.Scope}}</span>{{end}}</p>
{{else}}<p class="empty">You are not a member of any organization, team or project.</p>{{end}}
<p>{{range $i, $m := .Memberships}}{{if $i}} · {{end}}{{$m.Path}} <span class="opt">{{t (print "profile.scope." $m.Scope)}}</span>{{end}}</p>
{{else}}<p class="empty">{{t "profile.no_memberships"}}</p>{{end}}
<h2>Your use and limits</h2>
<h2>{{t "selfuse.heading"}}</h2>
{{with .Own}}
<div class="tiles own-use">
<div class="tile"><div class="v">{{compact .Requests}}</div><div class="k">requests · 30 days</div></div>
<div class="tile"><div class="v">{{compact .Tokens}}</div><div class="k">tokens · 30 days</div></div>
<div class="tile"><div class="v">{{.Spend}}</div><div class="k">spend · 30 days</div></div>
<div class="tile"><div class="v">{{compact .Requests}}</div><div class="k">{{t "selfuse.requests_30d"}}</div></div>
<div class="tile"><div class="v">{{compact .Tokens}}</div><div class="k">{{t "selfuse.tokens_30d"}}</div></div>
<div class="tile"><div class="v">{{.Spend}}</div><div class="k">{{t "selfuse.spend_30d"}}</div></div>
</div>
{{with .Chart}}<div class="chart-wrap"><h3 class="chart-title">Tokens per day</h3>{{.}}</div>{{end}}
{{with .Chart}}<div class="chart-wrap"><h3 class="chart-title">{{t "selfuse.tokens_per_day"}}</h3>{{.}}</div>{{end}}
{{end}}
{{range .LimitsBy}}
<h3 class="limits-for">{{.Label}}{{with .Use}} <span class="opt">— {{compact .Requests}} requests, {{compact .Tokens}} tokens in 30 days</span>{{end}}</h3>
<h3 class="limits-for">{{.Label}}{{with .Use}} <span class="opt">{{t "selfuse.place_use" (compact .Requests) (compact .Tokens)}}</span>{{end}}</h3>
{{range .Budgets}}
<div class="budget{{if .Over}} over{{else if .Near}} near{{end}}">
<div class="budget-head"><span>{{.Label}}</span><span><b>{{.Used}}</b> of {{.Budget}} {{.Period}}</span></div>
<div class="budget-head"><span>{{.Label}}</span><span>{{th "selfuse.budget_of" .Used .Budget .Period}}</span></div>
<div class="budget-bar"><span style="width: {{.Percent}}%"></span></div>
</div>
{{end}}
{{if .Rows}}
<table class="rows compact narrow-table"><tr><th>Limit</th><th class="num">Value</th></tr>
<table class="rows compact narrow-table"><tr><th>{{t "selfuse.col.limit"}}</th><th class="num">{{t "selfuse.col.value"}}</th></tr>
{{range .Rows}}<tr><td>{{.Name}}</td><td class="num">{{.Value}}</td></tr>{{end}}</table>
{{else}}<p class="empty">No limits here.</p>{{end}}
{{else}}<p class="empty">Nowhere to spend yet.</p>{{end}}
<p class="note">Everything that applies, from the organization down to you: the
strictest value wins.</p>
{{else}}<p class="empty">{{t "selfuse.no_limits"}}</p>{{end}}
{{else}}<p class="empty">{{t "selfuse.nowhere"}}</p>{{end}}
<p class="note">{{t "selfuse.strictest"}}</p>
<h2>Password</h2>
{{if .PasswordChanged}}<p>Your password was changed. Any other browser you were
signed in on has been signed out.</p>{{end}}
<h2>{{t "profile.password"}}</h2>
{{if .PasswordChanged}}<p>{{t "profile.password_changed"}}</p>{{end}}
{{if .PasswordError}}<p class="error">{{.PasswordError}}</p>{{end}}
<form method="post" action="/profile/password" autocomplete="on" class="narrow">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<label>Current password<input type="password" name="current" required
<label>{{t "profile.current_password"}}<input type="password" name="current" required
autocomplete="current-password"></label>
<label>New password<input type="password" name="new" required
<label>{{t "profile.new_password"}}<input type="password" name="new" required
minlength="{{.MinPassword}}" autocomplete="new-password"></label>
<button type="submit">Change password</button>
<button type="submit">{{t "profile.change_password"}}</button>
</form>
<p class="note">Changing it signs out every other browser, which is the point if
you are changing it because something went wrong.</p>
<p class="note">{{t "profile.password_note"}}</p>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}
+20 -19
View File
@@ -1,46 +1,47 @@
{{define "admin_guard"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Guardrails · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "guard.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>Guardrails</h1>
<p class="sub">What the policies did, where, and by which rule — names and counts, never what was written.</p>
<h1>{{t "guard.title"}}</h1>
<p class="sub">{{t "guard.sub"}}</p>
{{template "nav" .Nav}}
</header>
{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
<h2>Policies</h2>
<h2>{{t "guard.policies"}}</h2>
{{if .Policies}}
<p>Defined in <code>server.yaml</code>: {{range $i, $p := .Policies}}{{if $i}}, {{end}}<code>{{$p}}</code>{{end}}.
Applied to every request: {{if .Default}}{{range $i, $p := .Default}}{{if $i}}, {{end}}<code>{{$p}}</code>{{end}}{{else}}none{{end}}.</p>
<p>{{th "guard.defined_in"}} {{range $i, $p := .Policies}}{{if $i}}, {{end}}<code>{{$p}}</code>{{end}}.
{{t "guard.applied"}} {{if .Default}}{{range $i, $p := .Default}}{{if $i}}, {{end}}<code>{{$p}}</code>{{end}}{{else}}{{t "common.none"}}{{end}}.</p>
{{if .Attachments}}
<table class="rows">
<tr><th>Scope</th><th>Who</th><th>Policy</th></tr>
<tr><th>{{t "guard.col.scope"}}</th><th>{{t "guard.col.who"}}</th><th>{{t "guard.col.policy"}}</th></tr>
{{range .Attachments}}<tr><td>{{.Scope}}</td><td>{{.Who}}</td><td><code>{{.Policy}}</code></td></tr>{{end}}
</table>
{{else}}<p class="sub">Attached nowhere else. <code>llmgw ctl guard attach project:ORG/TEAM/PROJECT POLICY</code> attaches one.</p>{{end}}
{{else}}<p class="sub">{{th "guard.attached_nowhere"}}</p>{{end}}
{{else}}
<p>No policies: add a <code>guard:</code> section to <code>server.yaml</code> (docs/guardrails.md).</p>
<p>{{th "guard.no_policies"}}</p>
{{end}}
<h2>What they did</h2>
<p class="sub">Over the last
<h2>{{t "guard.what_they_did"}}</h2>
<p class="sub">{{t "guard.over_last"}}
{{range .Windows}}{{if eq . $.Window}}<strong>{{.}}</strong>{{else}}<a href="?window={{.}}">{{.}}</a>{{end}} {{end}}</p>
{{if .Counts}}
<table class="rows">
<tr><th>Where</th><th>Action</th><th>Policy</th><th>Category</th><th class="num">Requests</th></tr>
{{range .Counts}}<tr><td>{{.Where}}</td><td>{{.Action}}</td><td><code>{{.Policy}}</code></td><td>{{.Category}}</td><td class="num">{{.Count}}</td></tr>{{end}}
<tr><th>{{t "guard.col.where"}}</th><th>{{t "guard.col.action"}}</th><th>{{t "guard.col.policy"}}</th><th>{{t "guard.col.category"}}</th><th class="num">{{t "guard.col.requests"}}</th></tr>
{{range .Counts}}<tr><td>{{.Where}}</td><td>{{.Action}}</td><td><code>{{.Policy}}</code></td><td>{{.Category}}</td><td class="num">{{count .Count}}</td></tr>{{end}}
</table>
{{else}}<p class="sub">Nothing flagged, redacted or blocked.</p>{{end}}
{{else}}<p class="sub">{{t "guard.nothing"}}</p>{{end}}
{{if .Events}}
<h2>Recent</h2>
<h2>{{t "guard.recent"}}</h2>
<table class="rows">
<tr><th>When (UTC)</th><th>Request</th><th>Where</th><th>Model</th><th>Action</th><th>Policy</th><th>Category</th><th class="num">Status</th><th class="num">ms</th></tr>
<tr><th>{{t "guard.col.when"}}</th><th>{{t "guard.col.request"}}</th><th>{{t "guard.col.where"}}</th><th>{{t "guard.col.model"}}</th><th>{{t "guard.col.action"}}</th><th>{{t "guard.col.policy"}}</th><th>{{t "guard.col.category"}}</th><th class="num">{{t "guard.col.status"}}</th><th class="num">ms</th></tr>
{{range .Events}}<tr><td>{{.At}}</td><td><code>{{.RequestID}}</code></td><td>{{.Where}}</td><td>{{.Model}}</td>
<td>{{.Action}}</td><td><code>{{.Policy}}</code></td><td>{{.Category}}</td><td class="num">{{.Status}}</td><td class="num">{{.Ms}}</td></tr>{{end}}
<td>{{.Action}}</td><td><code>{{.Policy}}</code></td><td>{{.Category}}</td><td class="num">{{.Status}}</td><td class="num">{{count .Ms}}</td></tr>{{end}}
</table>
<p class="sub"><code>llmgw ctl request ID</code> says what else a request went through.</p>
<p class="sub">{{th "guard.request_hint"}}</p>
{{end}}
<footer><span>llmgw {{.Version}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}
+200 -217
View File
@@ -1,21 +1,21 @@
{{define "admin_mirror"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
{{/* Only while something is downloading. A reload at any other time would
take with it whatever the operator was typing into the form below. */}}
{{if .RefreshSec}}<meta http-equiv="refresh" content="{{.RefreshSec}}">{{end}}
<script defer src="/_static/pages.js"></script>
<title>Mirror · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "mirror.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>Model mirror</h1>
<p class="sub">{{.Addr}}{{if .Unreachable}} did not answer{{end}}</p>
<h1>{{t "mirror.heading"}}</h1>
<p class="sub">{{.Addr}}{{if .Unreachable}} {{t "mirror.did_not_answer"}}{{end}}</p>
{{template "nav" .Nav}}
</header>
{{/* Links rather than a script's panels: each tab has an address, the back
button works, and only the one shown asks anything of the registry. */}}
<nav class="page-tabs" aria-label="Mirror">
<nav class="page-tabs" aria-label="{{t "mirror.tabs_aria"}}">
{{range .Tabs}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}{{with .Count}} <span class="count">{{.}}</span>{{end}}</a>
{{end}}</nav>
@@ -27,50 +27,47 @@
{{/* Said outright rather than shown as an empty table. A mirror holding
nothing and a mirror that is not there look the same in a list of zero
rows, and they call for opposite responses. */}}
<p class="note">Nothing was read from the mirror, so this page is not a list of
what it holds — it is not currently reachable at <code>{{.Addr}}</code>. Check
that <code>llmgw-mirror</code> is running there and that its management port is
reachable from this machine.</p>
<p class="note">{{th "mirror.unreachable" .Addr}}</p>
{{else}}
<div id="held">
{{if .Rows}}
<table class="rows compact held">
<tr><th>Model</th><th>Files</th><th class="num">Size</th><th>Runs</th><th></th></tr>
<tr><th>{{t "mirror.col.model"}}</th><th>{{t "mirror.col.files"}}</th><th class="num">{{t "mirror.col.size"}}</th><th>{{t "mirror.col.runs"}}</th><th></th></tr>
{{range .Groups}}{{$grouped := gt .Models 1}}
{{if $grouped}}
<tr class="model-group"><td colspan="5"><strong>{{.Family}}</strong>
<span class="opt">{{.Models}} models · {{bytes .Bytes}}</span>
{{if .Fetching}}<span class="tag new">downloading</span>{{end}}
{{if .Failed}}<span class="tag bad">failed</span>{{end}}
{{if .InUse}}<span class="tag">in use</span>{{end}}</td></tr>
<span class="opt">{{tn "mirror.group.models" .Models (bytes .Bytes)}}</span>
{{if .Fetching}}<span class="tag new">{{t "mirror.state.fetching"}}</span>{{end}}
{{if .Failed}}<span class="tag bad">{{t "mirror.state.failed"}}</span>{{end}}
{{if .InUse}}<span class="tag">{{t "mirror.group.in_use"}}</span>{{end}}</td></tr>
{{end}}
{{range .Rows}}{{$row0 := .}}<tr{{if $grouped}} class="model-version"{{end}}{{if .Detail}} class="open"{{end}}>
<td>{{if eq (printf "%s" .State) "present"}}<a class="open-held" aria-expanded="{{if .Detail}}true{{else}}false{{end}}"
href="/admin/mirror?tab=held{{if not .Detail}}&amp;open={{.Open}}{{end}}"><strong>{{.Model}}</strong></a>
{{else}}<strong>{{.Model}}</strong>{{end}} <span class="opt">{{.Version}}</span>
{{if eq .Kind "adapter"}}<span class="tag new" title="a LoRA adapter for {{.Arch}} models: attach it on a model's card">LoRA adapter</span>{{end}}</td>
<td class="file" title="{{.File}}{{if .SHA256}} · sha256 {{.SHA256}}{{end}}">
{{if gt .Files 1}}{{.Files}} files{{else}}<code>{{.File}}</code>{{end}}</td>
{{if eq .Kind "adapter"}}<span class="tag new" title="{{t "mirror.adapter_title" .Arch}}">{{t "mirror.adapter"}}</span>{{end}}</td>
<td class="file" title="{{if .SHA256}}{{t "mirror.file_sha" .File .SHA256}}{{else}}{{.File}}{{end}}">
{{if gt .Files 1}}{{tn "mirror.files" .Files}}{{else}}<code>{{.File}}</code>{{end}}</td>
<td class="num">
{{if eq (printf "%s" .State) "fetching"}}
{{if .HasPercent}}{{.Percent}}% of {{bytes .BytesTotal}}
{{if .HasPercent}}{{t "mirror.percent_of" .Percent (bytes .BytesTotal)}}
{{else}}{{/* No total came from upstream, so no percentage is invented —
a bar that guesses its own end and then stops moving is worse
than a number that only goes up. */}}
{{bytes .BytesDone}} so far{{end}}
{{t "mirror.so_far" (bytes .BytesDone)}}{{end}}
{{else if eq (printf "%s" .State) "failed"}}—
{{else}}{{bytes .BytesDone}}{{end}}
</td>
<td>
{{if eq (printf "%s" .State) "fetching"}}<span class="state state-starting">downloading</span>
{{else if eq (printf "%s" .State) "failed"}}<span class="state state-failed">failed</span>
{{if eq (printf "%s" .State) "fetching"}}<span class="state state-starting">{{t "mirror.state.fetching"}}</span>
{{else if eq (printf "%s" .State) "failed"}}<span class="state state-failed">{{t "mirror.state.failed"}}</span>
{{else if .UsedBy}}{{$row := .}}{{range .UsedBy}}<div class="use">
{{/* Named only when the catalog calls it something other than the
mirror does; otherwise it is the model column again. */}}
{{if ne .Name $row.Model}}<span class="as" title="{{.Name}}">as {{.Name}}</span>{{end}}
{{if .Declared}}<span class="mode-fixed" title="set in server.yaml, which is read again on every start">
{{- if eq .Mode "always"}}always on{{else if eq .Mode "ondemand"}}on demand{{else}}stopped{{end}} · yaml</span>
{{if ne .Name $row.Model}}<span class="as" title="{{.Name}}">{{t "mirror.use.as" .Name}}</span>{{end}}
{{if .Declared}}<span class="mode-fixed" title="{{t "mirror.use.yaml_title"}}">
{{- t "mirror.use.yaml" (t (print "mirror.mode." .Mode))}}</span>
{{else}}
{{/* Submitted on change: the choice is the action, and a second click
to confirm it would only be a way to forget to. */}}
@@ -78,22 +75,22 @@ reachable from this machine.</p>
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="model" value="{{.Name}}">
<input type="hidden" name="replicas" value="{{.Replicas}}">
<select name="mode" class="mode" data-autosubmit aria-label="How {{.Name}} runs">
<option value="always"{{if eq .Mode "always"}} selected{{end}}>always on</option>
<option value="ondemand"{{if eq .Mode "ondemand"}} selected{{end}}>on demand</option>
<option value="stopped"{{if eq .Mode "stopped"}} selected{{end}}>stopped</option>
<select name="mode" class="mode" data-autosubmit aria-label="{{t "mirror.use.mode_aria" .Name}}">
<option value="always"{{if eq .Mode "always"}} selected{{end}}>{{t "mirror.mode.always"}}</option>
<option value="ondemand"{{if eq .Mode "ondemand"}} selected{{end}}>{{t "mirror.mode.ondemand"}}</option>
<option value="stopped"{{if eq .Mode "stopped"}} selected{{end}}>{{t "mirror.mode.stopped"}}</option>
</select>
<noscript><button type="submit" class="link">set</button></noscript>
{{/* The button is in the catalogue: what a noscript holds is read as text, not markup. */}}<noscript>{{th "mirror.use.set"}}</noscript>
</form>
{{end}}
{{if .Fleet}}<span class="runs-now{{if or .Parked .Unplaceable}} bad{{else if lt .Ready .Wanted}} short{{end}}" title="replicas ready / wanted{{if .Starting}}, + starting{{end}}">
{{- if .Parked}}parked
{{- else if and (eq .Wanted 0) (eq .Ready 0)}}none running
{{if .Fleet}}<span class="runs-now{{if or .Parked .Unplaceable}} bad{{else if lt .Ready .Wanted}} short{{end}}" title="{{if .Starting}}{{t "mirror.use.runs_title_starting"}}{{else}}{{t "mirror.use.runs_title"}}{{end}}">
{{- if .Parked}}{{t "mirror.use.parked"}}
{{- else if and (eq .Wanted 0) (eq .Ready 0)}}{{t "mirror.use.none_running"}}
{{- else}}{{.Ready}}/{{.Wanted}}{{if .Starting}} +{{.Starting}}{{end}}{{end}}
{{- if .Unplaceable}} · no room{{end}}
{{- if .Unplaceable}} · {{t "mirror.use.no_room"}}{{end}}
{{- with .Nodes}} · {{range $i, $n := .}}{{if $i}}, {{end}}<span class="node" title="{{$n.Full}}">{{$n.Short}}</span>{{end}}{{end}}</span>{{end}}
</div>{{end}}
{{else}}<span class="opt">not served</span>{{end}}
{{else}}<span class="opt">{{t "mirror.not_served"}}</span>{{end}}
</td>
<td class="actions">
{{if eq (printf "%s" .State) "failed"}}
@@ -108,16 +105,16 @@ reachable from this machine.</p>
<input type="hidden" name="model" value="{{.Model}}">
<input type="hidden" name="version" value="{{.Version}}">
<input type="hidden" name="file" value="{{.File}}">
<button type="submit" class="link">retry</button>
<button type="submit" class="link">{{t "mirror.retry"}}</button>
</form>
{{else if eq (printf "%s" .State) "present"}}
<button type="button" class="link copy-url" data-url="{{.URL}}" title="{{.URL}}">copy address</button>
{{if and $.HasCatalog (not .UsedBy)}}{{if .Serve}}<a href="/admin/mirror">close</a>
{{else}}<a href="/admin/mirror?serve={{.Model}}:{{.Version}}">serve</a>{{end}}{{end}}
<button type="button" class="link copy-url" data-url="{{.URL}}" title="{{.URL}}">{{t "mirror.copy_address"}}</button>
{{if and $.HasCatalog (not .UsedBy)}}{{if .Serve}}<a href="/admin/mirror">{{t "mirror.close"}}</a>
{{else}}<a href="/admin/mirror?serve={{.Model}}:{{.Version}}">{{t "mirror.serve"}}</a>{{end}}{{end}}
<form method="post" action="/admin/mirror/{{.Model}}/{{.Version}}/remove"
class="inline" data-confirm="Remove {{.Model}}/{{.Version}} and every file in it?{{if .UsedBy}} It is in use by the catalog.{{end}}">
class="inline" data-confirm="{{if .UsedBy}}{{t "mirror.remove_confirm_in_use" .Model .Version}}{{else}}{{t "mirror.remove_confirm" .Model .Version}}{{end}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">remove</button>
<button type="submit" class="link danger">{{t "mirror.remove"}}</button>
</form>
{{end}}
</td>
@@ -129,39 +126,38 @@ reachable from this machine.</p>
<tr class="editing"><td colspan="5">
{{if .Refusal}}
{{/* Offered and then failing at placement is worse than not offered at all. */}}
<p class="error">Cannot be served as it stands: {{.Refusal}}.</p>
<p class="error">{{t "serve.refusal" .Refusal}}</p>
{{else}}
{{if .Warning}}<p class="note error-row">{{.Warning}}.</p>{{end}}
<form method="post" action="/admin/mirror/serve" class="serve-form" data-engine="{{.Engine}}" data-free-gpus="{{.FreeGPUs}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="source" value="{{.Source}}">
<input type="hidden" name="engine" value="{{.Engine}}">
<p class="sub">Serving <code>{{.Source}}</code> with <strong>{{.Engine}}</strong>.
{{if .Files}}The whole version, {{.Files}} files: the node fetches them into one
directory and the engine is given the directory.{{end}}</p>
<p class="sub">{{th "serve.serving" .Source .Engine}}
{{if .Files}}{{t "serve.whole_version" .Files}}{{end}}</p>
<div class="field-row">
<label>Name <input type="text" name="name" value="{{.Model}}" required></label>
<label>Aliases <span class="opt">comma separated</span>
<input type="text" name="aliases" placeholder="coder-7b"></label>
<label>Replicas <input type="text" name="replicas" value="1" required></label>
<label>Runs
<label>{{t "serve.name"}} <input type="text" name="name" value="{{.Model}}" required></label>
<label>{{t "serve.aliases"}} <span class="opt">{{t "serve.aliases_hint"}}</span>
<input type="text" name="aliases" placeholder="{{t "serve.aliases_example"}}"></label>
<label>{{t "serve.replicas"}} <input type="text" name="replicas" value="1" required></label>
<label>{{t "serve.runs"}}
<select name="mode">
<option value="always">always on</option>
<option value="ondemand">on demand: loads when asked, stops after 15 min unused</option>
<option value="always">{{t "mirror.mode.always"}}</option>
<option value="ondemand">{{t "serve.mode.ondemand"}}</option>
</select></label>
</div>
{{/* Reaches clients through /v1/models, which for most of them is the only
view of what this gateway serves. Nine names and nothing to choose
between them is a choice made by guesswork. */}}
<label>What it is for <span class="opt">shown to clients in /v1/models</span>
<label>{{t "serve.description"}} <span class="opt">{{t "serve.description_hint"}}</span>
<input type="text" name="description"
placeholder="general chat, strong Swedish; 14B, slower than the 8B"></label>
placeholder="{{t "serve.description_example"}}"></label>
<div class="field-row">
<label>Context a request may use
<label>{{t "serve.context_length"}}
<input type="text" name="context_length" value="4096" required></label>
<label>Output limit <span class="opt">optional</span>
<label>{{t "serve.max_output"}} <span class="opt">{{t "serve.optional"}}</span>
<input type="text" name="max_output" value="1024"></label>
<label>Engine context pool <span class="opt">divided across slots</span>
<label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>
</div>
{{/* Where it runs is a choice with consequences, so it is a choice: the
@@ -169,56 +165,68 @@ reachable from this machine.</p>
field stood here once, and reading as though naming a profile "gpu"
would put it on one, it put GGUF models on the CPU of GPU clusters. */}}
<div class="field-row">
<label>Runs on
<label>{{t "serve.placement"}}
<select name="placement">
{{range .Placements}}<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>{{end}}
</select></label>
<label>GPUs per replica <span class="opt">on one node</span>
<label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>
{{if eq .Engine "llamacpp"}}<label>Parallel requests <span class="opt">per replica</span>
{{if eq .Engine "llamacpp"}}<label>{{t "serve.parallel"}} <span class="opt">{{t "serve.parallel_hint"}}</span>
<input type="text" name="parallel" value="{{.Parallel}}" size="3"></label>{{end}}
</div>
{{if .FreeGPUs}}<label class="cpu-ok"><input type="checkbox" name="cpu_ok" value="yes">
Run on the CPU anyway: {{.FreeGPUs}} GPUs are free and would sit unused</label>{{end}}
{{t "serve.cpu_ok" .FreeGPUs}}</label>{{end}}
<div class="field-row">
<label>Needs RAM (GB) <span class="opt">for the context pool beside it</span>
<label>{{t "serve.memory"}} <span class="opt">{{t "serve.memory_hint"}}</span>
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}" required></label>
<label>CPU cores <span class="opt">reserved; the engine's threads</span>
<label>{{t "serve.cpu_cores"}} <span class="opt">{{t "serve.cpu_cores_hint"}}</span>
<input type="text" name="cpu_cores" value="{{.CPUCores}}"></label>
<label>Only nodes labelled <span class="opt">key / value, optional</span>
<input type="text" name="selector_key" placeholder="role" size="8">
<input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>
<label>{{t "serve.selector"}} <span class="opt">{{t "serve.selector_hint"}}</span>
<input type="text" name="selector_key" placeholder="{{t "serve.selector_key_example"}}" size="8">
<input type="text" name="selector_value" placeholder="{{t "serve.selector_value_example"}}" size="10"></label>
</div>
<p class="sub serve-summary" hidden></p>
<button type="submit">Add to the catalog</button>
<button type="submit">{{t "serve.add"}}</button>
</form>
<script>
// What it will run as, in one line, kept up to date as the form changes:
// engine, where, how many cards, memory, context and how it is shared.
// engine, where, how many cards, memory, context and how it is shared. The
// sentences are the ones the server describes a model with, in the page's
// language; fmt fills their %s and %d, numbered ones included.
(function () {
const f = document.currentScript.previousElementSibling;
if (!f || !f.classList.contains("serve-form")) return;
const T = {
cpu: {{t "serve.on_cpu"}}, gpu1: {{t "serve.on_gpu_one"}}, gpus: {{t "serve.on_gpus"}},
mem: {{t "serve.host_memory"}}, cores: {{t "serve.cores.other"}},
poolPar: {{t "serve.pool_parallel"}}, pool: {{t "serve.pool"}},
runAs: {{t "serve.will_run_as"}}, free: {{t "serve.free_gpus_warning"}},
};
const fmt = (s, ...a) => {
let i = 0;
return s.replace(/%(?:\[(\d+)\])?[sd]/g, (_, n) => { const k = n ? +n - 1 : i; i = k + 1; return String(a[k]); });
};
const out = f.querySelector(".serve-summary"), ok = f.querySelector(".cpu-ok");
const v = (n) => (f.elements[n] ? f.elements[n].value.trim() : "");
const engine = { llamacpp: "llama.cpp", vllm: "vLLM", sglang: "SGLang" }[f.dataset.engine] || f.dataset.engine;
function draw() {
const place = v("placement"), gpus = parseInt(v("gpus"), 10) || 1;
const onCPU = place === "cpu" || place === "";
let where = "on the CPU";
let where = fmt(T.cpu, engine);
if (!onCPU) {
const cls = place.replace(/^gpu:/, "");
where = gpus > 1 ? "on " + gpus + " " + cls + " GPUs of one node" : "on 1 GPU " + cls;
where = gpus > 1 ? fmt(T.gpus, engine, gpus, cls) : fmt(T.gpu1, engine, cls);
}
const parts = [engine + " " + where];
if (v("min_memory_gb")) parts.push(v("min_memory_gb") + " GB of host memory");
if (onCPU && v("cpu_cores")) parts.push(v("cpu_cores") + " cores");
const parts = [where];
if (v("min_memory_gb")) parts.push(fmt(T.mem, v("min_memory_gb")));
if (onCPU && v("cpu_cores")) parts.push(fmt(T.cores, v("cpu_cores")));
const pool = parseInt(v("ctx_size"), 10), par = parseInt(v("parallel"), 10);
if (pool && par) parts.push("context pool " + pool + " over " + par + " parallel requests (" + Math.floor(pool / par) + " each)");
else if (pool) parts.push("context pool " + pool);
out.textContent = "Will run as: " + parts.join(", ") + ".";
if (pool && par) parts.push(fmt(T.poolPar, pool, par, Math.floor(pool / par)));
else if (pool) parts.push(fmt(T.pool, pool));
out.textContent = fmt(T.runAs, parts.join(", "));
const warn = onCPU && +f.dataset.freeGpus > 0;
out.classList.toggle("error", warn);
if (warn) out.textContent += " " + f.dataset.freeGpus + " GPUs are free; on the CPU this runs many times slower.";
if (warn) out.textContent += " " + fmt(T.free, f.dataset.freeGpus);
if (ok) ok.hidden = !onCPU;
out.hidden = false;
}
@@ -227,57 +235,45 @@ reachable from this machine.</p>
draw();
})();
</script>
<p class="note">The context pool is what the engine is given; llama.cpp divides
it across its parallel slots, so the number one request may use is the smaller
one above.</p>
<p class="note">Memory: {{.MemoryBasis}}. Change the context pool and this changes
with it, by the KV cache's share — an estimate, and editable. It is filled in rather than
left blank because a CPU profile has no card to be short of: with no figure,
every ready node is a candidate and the first one picked is whichever the
scheduler reached, not whichever can hold the model. A node label does the same
job by name if you would rather pin it.</p>
<p class="note">{{t "serve.pool_note"}}</p>
<p class="note">{{t "serve.memory_note" .MemoryBasis}}</p>
{{end}}
</td></tr>
{{end}}
{{end}}
{{end}}
</table>
<p class="note">“copy address” gives what a catalog entry's <code>source</code>
should say — the file, or for a version of several files the directory, with its
trailing slash. It is rendered rather than assembled by hand because the mirror
and the catalog have to spell a file the same way, which is also how this page
knows which models use it.</p>
<p class="note">{{th "mirror.copy_note"}}</p>
<script>
document.addEventListener('click', function (e) {
var b = e.target.closest('.copy-url');
if (!b) return;
var done = function (t) { var o = b.textContent; b.textContent = t; setTimeout(function () { b.textContent = o; }, 1500); };
if (navigator.clipboard) {
navigator.clipboard.writeText(b.dataset.url).then(function () { done('copied'); },
function () { window.prompt('Address', b.dataset.url); });
} else { window.prompt('Address', b.dataset.url); }
});
(function () {
var copied = {{t "common.copied"}}, address = {{t "mirror.address_prompt"}};
document.addEventListener('click', function (e) {
var b = e.target.closest('.copy-url');
if (!b) return;
var done = function (t) { var o = b.textContent; b.textContent = t; setTimeout(function () { b.textContent = o; }, 1500); };
if (navigator.clipboard) {
navigator.clipboard.writeText(b.dataset.url).then(function () { done(copied); },
function () { window.prompt(address, b.dataset.url); });
} else { window.prompt(address, b.dataset.url); }
});
})();
</script>
{{else}}<p class="empty">The mirror is reachable and holds nothing yet.
{{if .HasRegistry}}<a href="/admin/mirror?tab=explore">Explore Hugging Face</a> for a model, or
{{end}}<a href="/admin/mirror?tab=fetch">fetch one by URL</a>.</p>{{end}}
{{else}}<p class="empty">{{if .HasRegistry}}{{th "mirror.empty_explore"}}{{else}}{{th "mirror.empty"}}{{end}}</p>{{end}}
</div>
{{end}}
{{end}}
{{if and (eq .Tab "explore") .HasRegistry}}
{{if not .Repo}}{{with .Explore}}
<nav class="pills" aria-label="Which models">{{range .Views}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}</a>{{end}}</nav>
<nav class="pills" aria-label="{{t "explore.views_aria"}}">{{range .Views}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}</a>{{end}}</nav>
{{if eq .View "adapters"}}
<p class="note">{{if .Of}}LoRA adapters made for <code>{{.Of}}</code> ·
<a href="/admin/mirror?tab=explore&amp;view=adapters">all adapters</a>{{else}}LoRA adapters for any model.
From a held model's card, <em>find adapters</em> lists those made for it.{{end}}
An adapter runs on the model it was made for, and only as a GGUF adapter file;
open one to see which of its files are.</p>
<p class="note">{{if .Of}}{{th "explore.adapters_for" .Of}}
<a href="/admin/mirror?tab=explore&amp;view=adapters">{{t "explore.all_adapters"}}</a>{{else}}{{th "explore.adapters_any"}}{{end}}
{{t "explore.adapters_note"}}</p>
{{else}}
<div class="explore-filters">
<nav class="pills small" aria-label="Task">{{range .Tasks}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="true"{{end}}>{{.Label}}</a>{{end}}</nav>
<nav class="pills small" aria-label="Size">{{range .Sizes}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="true"{{end}}>{{.Label}}</a>{{end}}</nav>
<nav class="pills small" aria-label="{{t "explore.task_aria"}}">{{range .Tasks}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="true"{{end}}>{{.Label}}</a>{{end}}</nav>
<nav class="pills small" aria-label="{{t "explore.size_aria"}}">{{range .Sizes}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="true"{{end}}>{{.Label}}</a>{{end}}</nav>
</div>
{{end}}
{{end}}
@@ -288,9 +284,9 @@ document.addEventListener('click', function (e) {
<input type="hidden" name="size" value="{{.Size}}">
{{if .Of}}<input type="hidden" name="of" value="{{.Of}}">{{end}}
<input type="hidden" name="x" value="1">{{end}}
<input type="search" name="q" value="{{.Query}}" placeholder="Search Hugging Face — qwen coder, gemma, bge…" aria-label="Search Hugging Face">
<label class="check"><input type="checkbox" name="here" value="1" data-autosubmit{{if .Explore}}{{if .Explore.Here}} checked{{end}}{{end}}> only what these nodes can run</label>
<button type="submit">Search</button>
<input type="search" name="q" value="{{.Query}}" placeholder="{{t "explore.search_placeholder"}}" aria-label="{{t "explore.search_aria"}}">
<label class="check"><input type="checkbox" name="here" value="1" data-autosubmit{{if .Explore}}{{if .Explore.Here}} checked{{end}}{{end}}> {{t "explore.only_here"}}</label>
<button type="submit">{{t "explore.search"}}</button>
</form>
{{end}}
@@ -298,24 +294,24 @@ document.addEventListener('click', function (e) {
{{/* Its own line, not the page's error: the registry is somebody else's server
and it being down says nothing about the mirror, which is what this page is
for and has to keep working when Hugging Face does not. */}}
<p class="note error-row">The registry: {{.RegistryErr}}</p>
<p class="note error-row">{{t "explore.registry_err" .RegistryErr}}</p>
{{end}}
{{if not .Repo}}{{with .Explore}}
{{if .Rows}}
<table class="rows explore">
<tr><th>Repository</th><th class="num">Size</th><th>Architecture</th><th class="num">Context</th>
<th class="num">Downloads</th><th class="num">Likes</th><th class="num">Updated</th></tr>
<tr><th>{{t "explore.col.repo"}}</th><th class="num">{{t "explore.col.size"}}</th><th>{{t "explore.col.arch"}}</th><th class="num">{{t "explore.col.context"}}</th>
<th class="num">{{t "explore.col.downloads"}}</th><th class="num">{{t "explore.col.likes"}}</th><th class="num">{{t "explore.col.updated"}}</th></tr>
{{range .Rows}}<tr class="explore-row">
<td><a href="{{.Href}}" class="open-repo" data-repo="{{.ID}}" aria-expanded="false"><code>{{.ID}}</code></a>
{{if .New}}<span class="tag new">new</span>{{end}}{{if .Gated.Is}}<span class="tag">gated</span>{{end}}
{{if or .BaseModel .License}}<span class="from" title="{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}">{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}</span>{{end}}</td>
{{if .New}}<span class="tag new">{{t "explore.new"}}</span>{{end}}{{if .Gated.Is}}<span class="tag">{{t "explore.gated"}}</span>{{end}}
{{if or .BaseModel .License}}<span class="from" title="{{with .BaseModel}}{{t "explore.from" .}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}">{{with .BaseModel}}{{t "explore.from" .}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}</span>{{end}}</td>
<td class="num">{{.ParamsLabel}}</td>
<td>{{if .Architecture}}<span{{if not .Verdict.Servable}} class="cannot"{{end}} title="{{.Verdict.Reason}}">{{.Architecture}}{{if .Verdict.Checked}}{{if .Verdict.Servable}} ✓{{else}} ✗{{end}}{{end}}</span>{{else}}—{{end}}</td>
<td class="num">{{.ContextLabel}}</td>
<td class="num">{{compact .Downloads}}</td>
<td class="num">{{compact .Likes}}</td>
<td class="num">{{.Age}}</td>
<td class="num">{{.Updated}}</td>
</tr>{{end}}
</table>
<script>
@@ -323,6 +319,12 @@ document.addEventListener('click', function (e) {
// list is what somebody is choosing from, and losing it to read one card
// means finding their place again for the next. The link still goes to the
// repository's own page when this does not run, or with a modifier key.
const exploreText = {
loading: {{t "explore.js.loading"}}, loadFailed: {{t "explore.js.load_failed"}},
starting: {{t "explore.js.starting"}}, refused: {{t "explore.js.refused"}},
follow: {{t "explore.js.follow"}}, started: {{t "explore.js.started"}},
unreachable: {{t "explore.js.unreachable"}},
};
document.querySelector("table.explore").addEventListener("click", async function (e) {
const a = e.target.closest("a.open-repo");
if (!a || e.metaKey || e.ctrlKey || e.shiftKey || e.button !== 0) return;
@@ -339,7 +341,7 @@ document.querySelector("table.explore").addEventListener("click", async function
detail.className = "repo-detail";
const td = document.createElement("td");
td.colSpan = row.children.length;
td.textContent = "Loading…";
td.textContent = exploreText.loading;
detail.appendChild(td);
row.after(detail);
row.classList.add("open");
@@ -352,7 +354,7 @@ document.querySelector("table.explore").addEventListener("click", async function
// stripped of anything that runs: the same HTML the full page shows.
td.innerHTML = text;
} catch (err) {
td.textContent = "Could not load it: " + err;
td.textContent = exploreText.loadFailed + " " + err;
}
});
@@ -367,7 +369,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
const btn = form.querySelector("button[type=submit]");
const was = btn.textContent;
btn.disabled = true;
btn.textContent = "Starting…";
btn.textContent = exploreText.starting;
let note = form.parentElement.querySelector(".fetch-note");
if (!note) {
note = document.createElement("div");
@@ -381,7 +383,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
});
const out = await resp.json().catch(() => ({ error: "(" + resp.status + ")" }));
if (!resp.ok) {
note.textContent = out.error || "refused";
note.textContent = out.error || exploreText.refused;
note.classList.add("bad");
btn.disabled = false;
btn.textContent = was;
@@ -391,25 +393,23 @@ document.querySelector("table.explore").addEventListener("submit", async functio
note.textContent = "✓ " + out.message + " — ";
const link = document.createElement("a");
link.href = "/admin/mirror?tab=held";
link.textContent = "follow it under Held";
link.textContent = exploreText.follow;
note.appendChild(link);
btn.textContent = "Started";
btn.textContent = exploreText.started;
} catch (err) {
note.textContent = "Could not reach the gateway: " + err;
note.textContent = exploreText.unreachable + " " + err;
note.classList.add("bad");
btn.disabled = false;
btn.textContent = was;
}
});
</script>
{{if .Hidden}}<p class="note">{{.Hidden}} more {{if eq .Hidden 1}}is{{else}}are{{end}} hidden: their architecture is one the nodes' llama.cpp cannot load.
Untick “only what these nodes can run” to see them.</p>{{end}}
<p class="note">A gated repository needs its terms accepted with a Hugging Face
account, and this browses anonymously — the files will list, and fetching one
will fail at the mirror. Download it by hand and put it in the mirror
directory, or give the mirror a token.</p>
{{if .Hidden}}<p class="note">{{tn "explore.hidden" .Hidden}}
{{t "explore.untick"}}</p>{{end}}
<p class="note">{{t "explore.gated_note"}}</p>
{{else if not $.RegistryErr}}
<p class="empty">{{if $.Query}}Nothing matched “{{$.Query}}”{{else}}Nothing to list{{end}}{{if .Hidden}} that these nodes can run — {{.Hidden}} {{if eq .Hidden 1}}result was{{else}}results were{{end}} hidden for an architecture llama.cpp cannot load{{end}}.</p>
<p class="empty">{{if $.Query}}{{if .Hidden}}{{tn "explore.empty.query_hidden" .Hidden $.Query}}{{else}}{{t "explore.empty.query" $.Query}}{{end}}
{{- else}}{{if .Hidden}}{{tn "explore.empty.hidden" .Hidden}}{{else}}{{t "explore.empty.nothing"}}{{end}}{{end}}</p>
{{end}}
{{end}}{{end}}
@@ -417,65 +417,61 @@ directory, or give the mirror a token.</p>
{{end}}
{{if eq .Tab "fetch"}}
<p class="note">By hand, for anything the search does not reach.</p>
<p class="note">{{t "mirror.fetch.intro"}}</p>
<form method="post" action="/admin/mirror/fetch">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<input type="hidden" name="tab" value="fetch">
<label>Source URL
<label>{{t "mirror.fetch.source"}}
<input type="text" name="source" required
placeholder="https://huggingface.co/…/resolve/main/model.gguf"></label>
placeholder="{{t "mirror.fetch.source_example"}}"></label>
<div class="field-row">
<label>Model <input type="text" name="model" required placeholder="qwen2.5-coder-7b"></label>
<label>Version <input type="text" name="version" required placeholder="2026-09"></label>
<label>File <span class="opt">optional</span>
<input type="text" name="file" placeholder="taken from the URL"></label>
<label>{{t "mirror.fetch.model"}} <input type="text" name="model" required placeholder="{{t "mirror.fetch.model_example"}}"></label>
<label>{{t "mirror.fetch.version"}} <input type="text" name="version" required placeholder="2026-09"></label>
<label>{{t "mirror.fetch.file"}} <span class="opt">{{t "serve.optional"}}</span>
<input type="text" name="file" placeholder="{{t "mirror.fetch.file_hint"}}"></label>
</div>
<label class="inline-check"><input type="checkbox" name="force" value="1">
Fetch anyway <span class="opt">even if the nodes' llama.cpp cannot load it</span></label>
<button type="submit">Start download</button>
{{t "mirror.fetch.force"}} <span class="opt">{{t "mirror.fetch.force_hint"}}</span></label>
<button type="submit">{{t "mirror.fetch.start"}}</button>
</form>
<p class="note">The mirror answers as soon as the download is under way, not when
it finishes — a model is tens of gigabytes and nothing here holds a connection
open for that. The Held tab shows it arriving. A version is never
overwritten: fetching into one that exists is refused, so a new build is a new
version and the old weights stay where they are.</p>
<p class="note">{{t "mirror.fetch.note"}}</p>
{{end}}
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}
{{/* One repository: its card and its files. The whole page when opened by
address, and the row beneath a listing entry when opened from the list. */}}
{{define "mirror_repo"}}
{{if not .Inline}}<p><a href="{{.Back}}">← back to the list</a></p>
<h3>{{.Repo.ID}} <span class="opt">at {{.Repo.ShortRevision}}</span></h3>{{end}}
{{if not .Inline}}<p><a href="{{.Back}}">{{t "card.back"}}</a></p>
<h3>{{.Repo.ID}} <span class="opt">{{t "card.at" .Repo.ShortRevision}}</span></h3>{{end}}
<p class="repo-meta">
{{if .Inline}}<span>at <code>{{.Repo.ShortRevision}}</code></span>{{end}}
{{with .Repo.BaseModel}}<span>from {{range $i, $b := .}}{{if $i}}, {{end}}<code>{{$b}}</code>{{end}}</span>{{end}}
{{with .Repo.License}}<span>license {{.}}</span>{{end}}
{{if .Inline}}<span>{{th "card.at_code" .Repo.ShortRevision}}</span>{{end}}
{{with .Repo.BaseModel}}<span>{{t "card.from"}} {{range $i, $b := .}}{{if $i}}, {{end}}<code>{{$b}}</code>{{end}}</span>{{end}}
{{with .Repo.License}}<span>{{t "card.license" .}}</span>{{end}}
{{with .Repo.Languages}}<span>{{join .}}</span>{{end}}
<span><a href="{{.RepoURL}}" rel="noopener noreferrer">on Hugging Face ↗</a></span>
<span><a href="{{.RepoURL}}" rel="noopener noreferrer">{{t "card.on_registry"}}</a></span>
</p>
{{if .Card}}
<details class="model-card" open>
<summary>Model card</summary>
<summary>{{t "card.model_card"}}</summary>
<div class="card-body">{{.Card}}</div>
</details>
{{end}}
{{if .RepoSets}}
<table class="rows repo-files">
<tr><th>What</th><th class="num">Size</th><th>Published checksum</th><th></th></tr>
<tr><th>{{t "card.col.what"}}</th><th class="num">{{t "card.col.size"}}</th><th>{{t "card.col.checksum"}}</th><th></th></tr>
{{range .RepoSets}}<tr>
<td><code>{{.Name}}</code>
{{if .Shards}}<span class="opt">{{.Shards}} shards, fetched together</span>{{end}}
{{with .Companions}}<span class="opt" title="{{join .}}">with {{len .}} files the engine reads beside them</span>{{end}}
{{if eq .Kind "adapter"}}<span class="tag new" title="a LoRA adapter: it runs on a {{.Arch}} model it was made for, not on its own">LoRA adapter · {{.Arch}}</span>
{{else if eq .Kind "model"}}<span class="opt">model{{with .Arch}} · {{.}}{{end}}</span>{{end}}</td>
{{if .Shards}}<span class="opt">{{t "card.shards" .Shards}}</span>{{end}}
{{with .Companions}}<span class="opt" title="{{join .}}">{{t "card.companions" (len .)}}</span>{{end}}
{{if eq .Kind "adapter"}}<span class="tag new" title="{{t "card.adapter_title" .Arch}}">{{t "card.adapter" .Arch}}</span>
{{else if eq .Kind "model"}}<span class="opt">{{if .Arch}}{{t "card.model_arch" .Arch}}{{else}}{{t "card.model"}}{{end}}</span>{{end}}</td>
<td class="num">{{bytes .Bytes}}</td>
<td>{{with .SHA256}}<code title="{{.}}">{{slice . 0 12}}</code>{{else}}—{{end}}</td>
<td>
{{if not .Verdict.Servable}}
<span class="state state-failed" title="{{.Verdict.Reason}}">cannot be served</span>
<span class="state state-failed" title="{{.Verdict.Reason}}">{{t "card.cannot_serve"}}</span>
<span class="clip-inline" title="{{.Verdict.Reason}}">{{.Verdict.Reason}}</span>
{{else}}
{{if .Verdict.Architecture}}<span class="opt" title="{{.Verdict.Reason}}">{{.Verdict.Architecture}}{{if .Verdict.Checked}} ✓{{end}}</span>{{end}}
@@ -487,24 +483,16 @@ version and the old weights stay where they are.</p>
{{end}}
<input type="text" name="model" value="{{.Model}}" size="24">
<input type="text" name="version" value="{{.Version}}" size="10">
<button type="submit">Fetch</button>
<button type="submit">{{t "card.fetch"}}</button>
</form>
{{end}}
</td>
</tr>{{end}}
</table>
<p class="note">Only the weights are listed; a repository is mostly a README and
a tokenizer. A safetensors set is fetched with the files vLLM reads beside it —
config.json, the tokenizer, the shard index — and never with code. Shards of one file are one row and one button — a fifteen-shard
model is one decision, not fifteen. Two separate files are two rows because they
are usually alternatives: one quantisation or the other, not both.</p>
<p class="note">The source URLs are pinned to
<code>{{.Repo.ShortRevision}}</code> rather than to a branch, so what arrives is
what is listed here even if the branch moves in between — and a single file's
checksum above is what the registry publishes, to compare with what the mirror
computes once it has it.</p>
<p class="note">{{t "card.weights_note"}}</p>
<p class="note">{{th "card.pinned_note" .Repo.ShortRevision}}</p>
{{else}}
<p class="empty">No weight files in {{.Repo.ID}}.</p>
<p class="empty">{{t "card.no_weights" .Repo.ID}}</p>
{{end}}
{{end}}
@@ -512,54 +500,52 @@ computes once it has it.</p>
{{define "mirror_held_detail"}}{{$row := .Row}}{{$csrf := .CSRF}}{{$scopes := .D.Scopes}}
<div class="held-card">
<p class="repo-meta">
<span>{{if gt $row.Files 1}}{{$row.Files}} files{{else}}<code>{{$row.File}}</code>{{end}}</span>
<span>{{if gt $row.Files 1}}{{tn "mirror.files" $row.Files}}{{else}}<code>{{$row.File}}</code>{{end}}</span>
<span>{{bytes $row.BytesDone}}</span>
{{with $row.SHA256}}<span>sha256 <code title="{{.}}">{{slice . 0 12}}</code></span>{{end}}
{{with $row.SHA256}}<span translate="no">sha256 <code title="{{.}}">{{slice . 0 12}}</code></span>{{end}}
<span><code>{{$row.URL}}</code></span>
</p>
{{if not .D.Models}}
<p class="note">Not in the catalog: nothing runs this version.
{{if $row.Serve}}{{else}}Use <strong>serve</strong> on its row to put it into service.{{end}}</p>
<p class="note">{{t "mirror.detail.not_in_catalog"}}
{{if $row.Serve}}{{else}}{{th "mirror.detail.use_serve"}}{{end}}</p>
{{end}}
{{range .D.Models}}
<section class="held-model">
<h3>{{.Name}}
{{with .View}}<span class="opt">{{.Ready}} of {{.Desired}} ready{{if ne .Total .Ready}} · {{.Total}} in all{{end}}</span>
{{if .Parked}}<span class="tag bad">parked</span>{{else if .Unplaceable}}<span class="tag bad">does not fit</span>{{end}}{{end}}
{{with .View}}<span class="opt">{{t "mirror.detail.ready" .Ready .Desired}}{{if ne .Total .Ready}} · {{t "mirror.detail.in_all" .Total}}{{end}}</span>
{{if .Parked}}<span class="tag bad">{{t "mirror.use.parked"}}</span>{{else if .Unplaceable}}<span class="tag bad">{{t "mirror.detail.does_not_fit"}}</span>{{end}}{{end}}
</h3>
{{with .View.Reason}}<p class="note error-row">{{.}}</p>{{end}}
{{if .Replicas}}
<table class="rows compact replicas">
<tr><th>Replica</th><th>Node</th><th>State</th><th class="num">Age</th><th class="num">CPU</th><th class="num">Memory</th></tr>
<tr><th>{{t "mirror.replica.col.replica"}}</th><th>{{t "mirror.replica.col.node"}}</th><th>{{t "mirror.replica.col.state"}}</th><th class="num">{{t "mirror.replica.col.age"}}</th><th class="num">CPU</th><th class="num">{{t "mirror.replica.col.memory"}}</th></tr>
{{range .Replicas}}{{$rep := .}}{{range $i, $s := .Nodes}}<tr>
<td>{{if eq $i 0}}<code>{{$rep.ID}}</code>{{end}}</td>
<td><a href="/admin/nodes/{{$s.Node}}">{{$s.Node}}</a>{{if $s.MemoryGB}} <span class="opt">{{$s.MemoryGB}} GB</span>{{end}}</td>
<td>{{if eq $i 0}}<span class="state state-{{if eq $rep.State "ready"}}ready{{else if eq $rep.State "failed"}}failed{{else}}starting{{end}}">{{$rep.State}}</span>
{{with $rep.Restart}}<span class="opt">restart {{.}}</span>{{end}}{{end}}</td>
<td>{{if eq $i 0}}<span class="state state-{{if eq $rep.State "ready"}}ready{{else if eq $rep.State "failed"}}failed{{else}}starting{{end}}">{{$rep.StateText}}</span>
{{with $rep.Restart}}<span class="opt">{{t "mirror.replica.restart" .}}</span>{{end}}{{end}}</td>
<td class="num">{{if eq $i 0}}{{age $rep.Age}}{{end}}</td>
<td class="num">{{if $s.CPU}}{{printf "%.1f" $s.CPU}} cores{{else}}—{{end}}</td>
<td class="num">{{if $s.CPU}}{{t "mirror.replica.cores" (num $s.CPU 1)}}{{else}}—{{end}}</td>
<td class="num">{{if $s.MemBytes}}{{bytes $s.MemBytes}}{{else}}—{{end}}</td>
</tr>{{end}}{{end}}
</table>
{{else}}<p class="note">No replica runs now{{if eq .Mode "ondemand"}}: it starts when somebody asks for it{{end}}.</p>{{end}}
{{else}}<p class="note">{{if eq .Mode "ondemand"}}{{t "mirror.detail.none_running_ondemand"}}{{else}}{{t "mirror.detail.none_running"}}{{end}}</p>{{end}}
{{with .Headroom}}
{{if .Headroom}}
<p class="headroom">
{{if .Limited}}Room for at least {{.More}} more replicas.
{{else if .More}}Room for {{.More}} more now{{if .Nodes}}, on {{range $i, $n := .Nodes}}{{if $i}}, {{end}}{{join $n}}{{end}}{{end}}.
{{else}}No free room for another replica now.{{end}}
{{if .ByPreempting}}<br><span class="note">{{if .More}}And {{end}}{{.ByPreempting}} {{if .More}}more {{end}}by stopping replicas its class outranks: {{join .Preempts}}.</span>{{end}}
{{.RoomText}}
{{with .PreemptText}}<br><span class="note">{{.}}</span>{{end}}
</p>
{{if and (not .Limited) .Blocked}}<details class="blocked"><summary>Why not {{if .More}}more{{else}}another{{end}}</summary>
<ul>{{range .Blocked}}<li>{{.}}</li>{{end}}</ul></details>{{end}}
{{if and (not .Headroom.Limited) .Headroom.Blocked}}<details class="blocked"><summary>{{if .Headroom.More}}{{t "mirror.room.why_not_more"}}{{else}}{{t "mirror.room.why_not_another"}}{{end}}</summary>
<ul>{{range .Headroom.Blocked}}<li>{{.}}</li>{{end}}</ul></details>{{end}}
{{end}}
{{if not .Declared}}{{$mname := .Name}}
<div class="adapters">
<h4>LoRA adapters <a class="note" href="{{.FindAdapters}}">find adapters on Hugging Face →</a></h4>
<h4>{{t "mirror.adapters.heading"}} <a class="note" href="{{.FindAdapters}}">{{t "mirror.adapters.find"}}</a></h4>
{{if .Adapters}}<table class="compact rows">
<tr><th>Asked for as</th><th>File</th><th class="num">Scale</th><th></th></tr>
<tr><th>{{t "mirror.adapters.asked_as"}}</th><th>{{t "mirror.adapters.file"}}</th><th class="num">{{t "mirror.adapters.scale"}}</th><th></th></tr>
{{range .Adapters}}{{$a := .}}<tr>
<td><code>{{.Name}}</code></td>
<td class="clip" title="{{.Source}}">{{.Source}}</td>
@@ -570,9 +556,9 @@ computes once it has it.</p>
<input type="hidden" name="open" value="{{$row.Open}}">
<input type="hidden" name="op" value="remove">
<input type="hidden" name="name" value="{{$a.Name}}">
<button type="submit" class="link danger">remove</button></form></td>
<button type="submit" class="link danger">{{t "mirror.remove"}}</button></form></td>
</tr>{{end}}</table>
{{else}}<p class="note">None. A request for an adapter's name is served by this model's replicas with it applied.</p>{{end}}
{{else}}<p class="note">{{t "mirror.adapters.none"}}</p>{{end}}
{{if .AdapterChoices}}{{$m := .}}
<form method="post" action="/admin/mirror/adapters" class="adapter-form">
<input type="hidden" name="csrf" value="{{$csrf}}">
@@ -580,53 +566,50 @@ computes once it has it.</p>
<input type="hidden" name="open" value="{{$row.Open}}">
<input type="hidden" name="op" value="add">
<div class="field-row">
<label>Adapter file <select name="source" required data-suggest-into="name">
<label>{{t "mirror.adapters.adapter_file"}} <select name="source" required data-suggest-into="name">
{{range .AdapterChoices}}<option value="{{.URL}}" data-suggest="{{.Suggest}}">{{.Label}}</option>{{end}}
</select></label>
<label>Asked for as <input type="text" name="name" value="{{(index .AdapterChoices 0).Suggest}}" required></label>
<label>Scale <input type="text" name="scale" value="1" size="4"></label>
<button type="submit">Attach</button>
<label>{{t "mirror.adapters.asked_as"}} <input type="text" name="name" value="{{(index .AdapterChoices 0).Suggest}}" required></label>
<label>{{t "mirror.adapters.scale"}} <input type="text" name="scale" value="1" size="4"></label>
<button type="submit">{{t "mirror.adapters.attach"}}</button>
</div>
</form>{{else}}<p class="note">No held adapter file for this model's architecture. Fetch one from Hugging Face first.</p>{{end}}
</form>{{else}}<p class="note">{{t "mirror.adapters.no_choice"}}</p>{{end}}
</div>
{{end}}
{{if .Declared}}
<p class="note">Set in server.yaml, which is read again on every start: change its replicas there.</p>
<p class="note">{{t "mirror.detail.declared"}}</p>
{{else}}
<form method="post" action="/admin/mirror/replicas" class="replicas-form">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{.Name}}">
<input type="hidden" name="open" value="{{$row.Open}}">
<div class="field-row">
<label>Always running
<label>{{t "mirror.form.min"}}
<input type="number" name="min" value="{{.Min}}" min="0"{{if .MostNow}} max="{{.MostNow}}"{{end}} required></label>
<label>Up to, under load
<label>{{t "mirror.form.max"}}
<input type="number" name="max" value="{{.Max}}" min="0" max="64" required></label>
<label>Class
<label>{{t "mirror.form.class"}}
<select name="class">
<option value=""{{if eq .Class ""}} selected{{end}}>default ({{if eq .Mode "ondemand"}}opportunistic{{else}}standard{{end}})</option>
<option value="critical"{{if eq .Class "critical"}} selected{{end}}>critical</option>
<option value="standard"{{if eq .Class "standard"}} selected{{end}}>standard</option>
<option value="opportunistic"{{if eq .Class "opportunistic"}} selected{{end}}>opportunistic</option>
<option value=""{{if eq .Class ""}} selected{{end}}>{{if eq .Mode "ondemand"}}{{t "mirror.class.default" (t "mirror.class.opportunistic")}}{{else}}{{t "mirror.class.default" (t "mirror.class.standard")}}{{end}}</option>
<option value="critical"{{if eq .Class "critical"}} selected{{end}}>{{t "mirror.class.critical"}}</option>
<option value="standard"{{if eq .Class "standard"}} selected{{end}}>{{t "mirror.class.standard"}}</option>
<option value="opportunistic"{{if eq .Class "opportunistic"}} selected{{end}}>{{t "mirror.class.opportunistic"}}</option>
</select></label>
{{if $scopes}}{{$owner := .Owner}}<label>Owner
{{if $scopes}}{{$owner := .Owner}}<label>{{t "mirror.form.owner"}}
<select name="owner">
<option value=""{{if eq $owner ""}} selected{{end}}>shared</option>
<option value=""{{if eq $owner ""}} selected{{end}}>{{t "mirror.form.shared"}}</option>
{{range $scopes}}<option value="{{.Key}}"{{if eq .Key $owner}} selected{{end}}>{{.Name}}</option>{{end}}
</select></label>{{end}}
</div>
<div class="field-row">
<label>Memory GB <span class="opt">per replica</span>
<label>{{t "mirror.form.memory"}} <span class="opt">{{t "mirror.form.per_replica"}}</span>
<input type="number" name="memory" value="{{.MemoryGB}}" min="0"></label>
<label>Cores <span class="opt">per replica, its threads</span>
<label>{{t "mirror.form.cores"}} <span class="opt">{{t "mirror.form.cores_hint"}}</span>
<input type="number" name="cores" value="{{.CPUCores}}" min="0"></label>
<button type="submit">Save</button>
<button type="submit">{{t "common.save"}}</button>
</div>
<p class="note">{{if .MostNow}}At most {{.MostNow}} can always run now{{if .Headroom}}{{if .Headroom.ByPreempting}}, counting what its class may stop to make room{{end}}{{end}}. {{end}}0 always running with a
higher limit loads it when asked for; 0 and 0 stops it. A replica is added under load when requests wait, and
taken away after ten quiet minutes. A higher class may stop replicas of lower ones to make room; memory and
cores apply to replicas started after the change.</p>
<p class="note">{{if .MostNow}}{{if and .Headroom .Headroom.ByPreempting}}{{t "mirror.form.most_now_preempting" .MostNow}}{{else}}{{t "mirror.form.most_now" .MostNow}}{{end}} {{end}}{{t "mirror.form.note"}}</p>
</form>
{{end}}
</section>
+53 -53
View File
@@ -3,25 +3,25 @@
links work as plain navigation; htmx, when it loads, swaps only #people,
so a change does not jump the page. */}}
{{define "admin_people"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>People · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "people.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false,"allowEval":false}'>
<script src="/_static/htmx-2.0.11.min.js" defer></script>
</head><body><div class="wrap">
<header>
<h1>People</h1>
<p class="sub">Organizations, teams, projects and the people in them: what each may use, and what it used</p>
<h1>{{t "people.title"}}</h1>
<p class="sub">{{t "people.sub"}}</p>
{{template "nav" .Nav}}
</header>
<div id="people" hx-boost="true" hx-target="#people" hx-select="#people"
hx-swap="outerHTML" hx-push-url="false">
<nav class="page-tabs" aria-label="People">
{{if .HasPeople}}<a href="/admin/users" hx-push-url="true"{{if eq .Tab "people"}} aria-current="page"{{end}}>Organizations &amp; people{{if .Pending}} <span class="tag">{{.Pending}} waiting</span>{{end}}</a>{{end}}
{{if .HasLimits}}<a href="/admin/quotas" hx-push-url="true"{{if eq .Tab "limits"}} aria-current="page"{{end}}>Limits</a>{{end}}
{{if .HasUsage}}<a href="/admin/usage" hx-push-url="true"{{if eq .Tab "usage"}} aria-current="page"{{end}}>Usage</a>{{end}}
<nav class="page-tabs" aria-label="{{t "people.title"}}">
{{if .HasPeople}}<a href="/admin/users" hx-push-url="true"{{if eq .Tab "people"}} aria-current="page"{{end}}>{{t "people.tab.people"}}{{if .Pending}} <span class="tag">{{t "people.waiting" .Pending}}</span>{{end}}</a>{{end}}
{{if .HasLimits}}<a href="/admin/quotas" hx-push-url="true"{{if eq .Tab "limits"}} aria-current="page"{{end}}>{{t "people.tab.limits"}}</a>{{end}}
{{if .HasUsage}}<a href="/admin/usage" hx-push-url="true"{{if eq .Tab "usage"}} aria-current="page"{{end}}>{{t "people.tab.usage"}}</a>{{end}}
</nav>
{{if .Users}}
@@ -34,15 +34,15 @@
{{end}}
</div>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div>
<script>
// Search the account list. Outside #people so what was typed survives a swap,
// and applied again after every swap.
(function () {
var box = document.createElement('input');
box.type = 'search'; box.placeholder = 'Find a person by email, name or project';
box.className = 'search'; box.setAttribute('aria-label', 'Find a person');
box.type = 'search'; box.placeholder = {{t "people.search.placeholder"}};
box.className = 'search'; box.setAttribute('aria-label', {{t "people.search.label"}});
function apply() {
var q = box.value.trim().toLowerCase();
document.querySelectorAll('#account-list tr[data-q]').forEach(function (tr) {
@@ -72,84 +72,84 @@
{{/* One branch or person: its limits, its capacity and its use, together. */}}
{{define "people-detail"}}{{$d := .}}{{$sel := printf "%s:%s" .Kind .ID}}
<div class="pd-head">
<span class="q-kind">{{if eq (printf "%s" .Kind) "user"}}person{{else}}{{.Kind}}{{end}}</span>
<span class="q-kind">{{t (print "people.kind." .Kind)}}</span>
<h2>{{.Name}}</h2>
{{if and .Path (ne .Path .Name)}}<p class="pd-path">{{.Path}}</p>{{end}}
<a href="/admin/users" class="pd-close" aria-label="Close">×</a>
<a href="/admin/users" class="pd-close" aria-label="{{t "people.close"}}">×</a>
</div>
{{with .Memberships}}<p class="pd-sub">In {{range $i, $m := .}}{{if $i}}, {{end}}<a href="/admin/users?sel={{$m.Scope}}:{{$m.ScopeID}}">{{$m.Path}}</a>{{end}}</p>
{{else}}{{if eq (printf "%s" $d.Kind) "user"}}<p class="pd-sub">On their own: in no organization, team or project.</p>{{end}}{{end}}
{{with .Memberships}}<p class="pd-sub">{{t "people.in"}} {{range $i, $m := .}}{{if $i}}, {{end}}<a href="/admin/users?sel={{$m.Scope}}:{{$m.ScopeID}}">{{$m.Path}}</a>{{end}}</p>
{{else}}{{if eq (printf "%s" $d.Kind) "user"}}<p class="pd-sub">{{t "people.on_own"}}</p>{{end}}{{end}}
{{if or .Limits .Person}}
<h3>Limits</h3>
{{if not .BudgetsBind}}<p class="note">No model has a price, so budgets stay at zero; rate and token limits apply.</p>{{end}}
<h3>{{t "people.limits"}}</h3>
{{if not .BudgetsBind}}<p class="note">{{t "people.budgets_unpriced"}}</p>{{end}}
{{with .Limits}}
<dl class="pd-limits">
<dt>Requests/min</dt><dd>{{template "quota-value" dict "Own" .RPM "Up" .RPMUp "From" .RPMFrom}}</dd>
<dt>Tokens/min</dt><dd>{{template "quota-value" dict "Own" .TPM "Up" .TPMUp "From" .TPMFrom}}</dd>
<dt>Budget</dt><dd>{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}{{else}}{{template "quota-value" dict "Own" "" "Up" .BudgetUp "From" .BudgetFrom}}{{end}}</dd>
<dt>Spent, {{$d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{.Percent}}%</span>{{end}}</dd>
<dt>Models</dt><dd>{{if .Models}}only <code>{{.Models}}</code>{{else if .ModelsFrom}}limited by {{.ModelsFrom}}{{else}}every model{{end}}</dd>
{{if .Member}}<dt>Each member</dt><dd>{{.Member}}</dd>{{end}}
<dt>{{t "people.rpm"}}</dt><dd>{{template "quota-value" dict "Own" .Shown.RPM "Up" .RPMUp "From" .RPMFrom}}</dd>
<dt>{{t "people.tpm"}}</dt><dd>{{template "quota-value" dict "Own" .Shown.TPM "Up" .TPMUp "From" .TPMFrom}}</dd>
<dt>{{t "people.budget"}}</dt><dd>{{if .Budget}}{{.Shown.Budget}}{{if .Period}}<span class="q-per">/{{.Shown.Period}}</span>{{end}}{{else}}{{template "quota-value" dict "Up" .BudgetUp "From" .BudgetFrom}}{{end}}</dd>
<dt>{{t "people.spent" $d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}</dd>
<dt>{{t "people.models"}}</dt><dd>{{if .Models}}{{th "people.models_only" .Models}}{{else if .ModelsFrom}}{{t "people.models_limited_by" .ModelsFrom}}{{else}}{{t "people.models_every"}}{{end}}</dd>
{{if .Member}}<dt>{{t "people.each_member"}}</dt><dd>{{.Member}}</dd>{{end}}
</dl>
{{if .Shadowed}}<p class="q-sub q-warn">{{.Shadowed}}</p>{{end}}
<details class="edit"><summary>{{if .HasAny}}change{{else}}set{{end}} limits</summary>
<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
{{template "quota-form" (dict "Scope" (printf "%s" .Scope) "ID" .ID "CSRF" $d.CSRF "Sel" $sel
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
"Title" (printf "Quota for %s" .Crumb)
"Note" "Shared by everything beneath it.")}}
"Title" (t "people.quota_for" .Crumb)
"Note" (t "people.note.shared"))}}
</details>
<details class="edit"><summary>limit each member</summary>
<details class="edit"><summary>{{t "people.limit_each_member"}}</summary>
{{template "quota-form" (dict "Scope" (printf "%s-member" .Scope) "ID" .ID "F" .MemberRaw "CSRF" $d.CSRF "Sel" $sel
"Title" (printf "Each member of %s, separately" .Crumb)
"Note" "Every member gets this much here, counted apart from the others.")}}
"Title" (t "people.each_member_of" .Crumb)
"Note" (t "people.note.each_member"))}}
</details>
{{end}}
{{with .Person}}
<dl class="pd-limits">
<dt>Requests/min</dt><dd>{{or .RPM "—"}}</dd>
<dt>Tokens/min</dt><dd>{{or .TPM "—"}}</dd>
<dt>Budget</dt><dd>{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}{{else}}—{{end}}</dd>
<dt>Spent, {{$d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{.Percent}}%</span>{{end}}</dd>
<dt>Models</dt><dd>{{if .Models}}only <code>{{.Models}}</code>{{else}}every model{{end}}</dd>
<dt>{{t "people.rpm"}}</dt><dd>{{or .Shown.RPM "—"}}</dd>
<dt>{{t "people.tpm"}}</dt><dd>{{or .Shown.TPM "—"}}</dd>
<dt>{{t "people.budget"}}</dt><dd>{{if .Budget}}{{.Shown.Budget}}{{if .Period}}<span class="q-per">/{{.Shown.Period}}</span>{{end}}{{else}}—{{end}}</dd>
<dt>{{t "people.spent" $d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}</dd>
<dt>{{t "people.models"}}</dt><dd>{{if .Models}}{{th "people.models_only" .Models}}{{else}}{{t "people.models_every"}}{{end}}</dd>
</dl>
<details class="edit"><summary>{{if .HasAny}}change{{else}}set{{end}} limits</summary>
<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
{{template "quota-form" (dict "Scope" "user" "ID" .ID "CSRF" $d.CSRF "Sel" $sel
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
"Title" (printf "%s, everywhere" .Email)
"Note" "Across every key this person holds, every project they use and the chat page.")}}
"Title" (t "people.everywhere" .Email)
"Note" (t "people.note.person"))}}
</details>
{{end}}
{{end}}
{{if .HasCapacity}}
<h3>Capacity</h3>
<h3>{{t "people.capacity"}}</h3>
{{with .Capacity}}
<dl class="pd-limits">
<dt>GPUs</dt><dd>{{or .GPUs "—"}}</dd>
<dt>Cores</dt><dd>{{or .Cores "—"}}</dd>
<dt>Memory GB</dt><dd>{{or .Memory "—"}}</dd>
<dt>{{t "people.cores"}}</dt><dd>{{or .Cores "—"}}</dd>
<dt>{{t "people.memory_gb"}}</dt><dd>{{or .Memory "—"}}</dd>
</dl>
<p class="note">Held of guaranteed (at most the limit). <a href="/admin/quotas?cap={{$sel}}#capacity">change</a></p>
{{else}}<p class="note">No capacity quota: its models share the cluster by their class alone.
<a href="/admin/quotas?cap={{$sel}}#capacity">set one</a></p>{{end}}
<p class="note">{{t "people.capacity_note"}} <a href="/admin/quotas?cap={{$sel}}#capacity">{{t "people.change"}}</a></p>
{{else}}<p class="note">{{t "people.no_capacity"}}
<a href="/admin/quotas?cap={{$sel}}#capacity">{{t "people.set_one"}}</a></p>{{end}}
{{end}}
{{if .HasUsage}}
<h3>Use, {{.UsageWindow}}</h3>
<h3>{{t "people.use" .UsageWindow}}</h3>
{{if .Totals.Name}}
<p class="pd-sub">{{compact .Totals.Requests}} requests · {{compact .Totals.Tokens}} tokens{{if .Priced}} · {{.Totals.Spend}} spent{{end}}{{if .Totals.Errors}} · {{.Totals.Errors}} errors{{end}}</p>
<p class="pd-sub">{{t "people.n_requests" (compact .Totals.Requests)}} · {{t "people.n_tokens" (compact .Totals.Tokens)}}{{if .Priced}} · {{t "people.spent_amount" .Totals.Spend}}{{end}}{{if .Totals.Errors}} · {{t "people.n_errors" (count .Totals.Errors)}}{{end}}</p>
{{if and .HasStats .Stats.Requests}}{{with .Stats}}
<dl class="pd-limits">
<dt>Error rate</dt><dd{{if ge .ErrorRate 5.0}} class="q-warn"{{end}}>{{pct1 .ErrorRate}}%</dd>
<dt>Refused by a limit</dt><dd{{if .Limited}} class="q-warn"{{end}}>{{if .Limited}}{{compact .Limited}}{{else}}none{{end}}</dd>
<dt>First token</dt><dd>p50 {{pctile .TTFT 50}} · p95 {{pctile .TTFT 95}} · p99 {{pctile .TTFT 99}}</dd>
<dt>Latency</dt><dd>p50 {{pctile .Latency 50}} · p95 {{pctile .Latency 95}} · p99 {{pctile .Latency 99}}</dd>
<dt>{{t "people.error_rate"}}</dt><dd{{if ge .ErrorRate 5.0}} class="q-warn"{{end}}>{{pct1 .ErrorRate}}%</dd>
<dt>{{t "people.refused"}}</dt><dd{{if .Limited}} class="q-warn"{{end}}>{{if .Limited}}{{compact .Limited}}{{else}}{{t "common.none"}}{{end}}</dd>
<dt>{{t "people.first_token"}}</dt><dd>p50 {{pctile .TTFT 50}} · p95 {{pctile .TTFT 95}} · p99 {{pctile .TTFT 99}}</dd>
<dt>{{t "people.latency"}}</dt><dd>p50 {{pctile .Latency 50}} · p95 {{pctile .Latency 95}} · p99 {{pctile .Latency 99}}</dd>
</dl>{{end}}{{end}}
{{template "tschart" dict "t" .Requests "title" "Requests" "note" ""}}
{{template "tschart" dict "t" .Tokens "title" "Tokens" "note" ""}}
{{if .Priced}}{{template "tschart" dict "t" .Spend "title" "Spend" "note" ""}}{{end}}
{{else}}<p class="empty">Nothing used in this window.</p>{{end}}
{{template "tschart" dict "t" .Requests "title" (t "people.chart.requests") "note" ""}}
{{template "tschart" dict "t" .Tokens "title" (t "people.chart.tokens") "note" ""}}
{{if .Priced}}{{template "tschart" dict "t" .Spend "title" (t "people.chart.spend") "note" ""}}{{end}}
{{else}}<p class="empty">{{t "people.nothing_used"}}</p>{{end}}
{{end}}
{{end}}
+85 -103
View File
@@ -6,10 +6,7 @@
prices every request costs nothing, so these numbers read as limits and
bind on nothing. Self-hosted models having no per-token price is ordinary;
showing somebody a budget without saying so is not. */}}
<p class="banner-warn" role="note">No model has a price, so nothing is charged and
budgets stay at zero however much is used. Rate and token limits still apply. Set
prices in the <code>pricing</code> block of the server configuration to make
budgets bind.</p>
<p class="banner-warn" role="note">{{th "quotas.unpriced"}}</p>
{{end}}
{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
@@ -20,42 +17,42 @@ budgets bind.</p>
one opens on that branch rather than asking somebody to name it again in a
box at the bottom. */}}
<table class="rows quota-tree">
<tr><th>Where</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
<th class="num">Budget</th><th>Spent</th><th></th></tr>
<tr><th>{{t "quotas.col.where"}}</th><th class="num">{{t "quotas.col.rpm"}}</th><th class="num">{{t "quotas.col.tpm"}}</th>
<th class="num">{{t "quotas.col.budget"}}</th><th>{{t "quotas.col.spent"}}</th><th></th></tr>
{{range .Rows}}
<tr class="q-depth-{{.Indent}}{{if .Editing}} editing{{end}}">
<td class="q-where">
<span class="q-name">{{.Path}}</span> <span class="q-kind">{{.Scope}}</span>
{{if .Models}}<div class="q-sub">only <code>{{.Models}}</code></div>
{{else if .ModelsFrom}}<div class="q-sub">models limited by {{.ModelsFrom}}</div>{{end}}
{{if .Member}}<div class="q-sub">each member here: {{.Member}}</div>{{end}}
<span class="q-name">{{.Path}}</span> <span class="q-kind">{{t (print "quotas.kind." .Scope)}}</span>
{{if .Models}}<div class="q-sub">{{th "quotas.only_models" .Models}}</div>
{{else if .ModelsFrom}}<div class="q-sub">{{t "quotas.models_from" .ModelsFrom}}</div>{{end}}
{{if .Member}}<div class="q-sub">{{t "quotas.each_member" .Member}}</div>{{end}}
{{if .Shadowed}}
{{/* A value that is written here and never reached. Resolution takes the
most restrictive along the chain, so without saying so the page would
show a number as though it applied. */}}
<div class="q-sub q-warn">{{.Shadowed}}</div>{{end}}
</td>
<td class="num">{{template "quota-value" dict "Own" .RPM "Up" .RPMUp "From" .RPMFrom}}</td>
<td class="num">{{template "quota-value" dict "Own" .TPM "Up" .TPMUp "From" .TPMFrom}}</td>
<td class="num">{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}
{{else}}{{template "quota-value" dict "Own" "" "Up" .BudgetUp "From" .BudgetFrom}}{{end}}</td>
<td class="num">{{template "quota-value" dict "Own" .Shown.RPM "Up" .RPMUp "From" .RPMFrom}}</td>
<td class="num">{{template "quota-value" dict "Own" .Shown.TPM "Up" .TPMUp "From" .TPMFrom}}</td>
<td class="num">{{if .Budget}}{{.Shown.Budget}}{{if .Period}}<span class="q-per">/{{.Shown.Period}}</span>{{end}}
{{else}}{{template "quota-value" dict "Up" .BudgetUp "From" .BudgetFrom}}{{end}}</td>
<td class="q-spent">
<span class="q-spend">{{.Spend}}</span>
{{if .HasPct}}<span class="q-meter{{if ge .Percent 90}} hot{{end}}" title="{{.Percent}}% of its budget"><i style="width:{{pct .Percent 100}}%"></i></span>
<span class="q-pct">{{.Percent}}%</span>{{end}}
{{if .HasPct}}<span class="q-meter{{if ge .Percent 90}} hot{{end}}" title="{{t "quotas.pct_of_budget" .Percent}}"><i style="width:{{pct .Percent 100}}%"></i></span>
<span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}
</td>
<td class="q-act">
{{if .Editing}}<a href="/admin/quotas">close</a>
{{else}}<a href="/admin/quotas?edit={{.Scope}}:{{.ID}}">{{if .HasAny}}change{{else}}set{{end}}</a>{{end}}
{{if .MemberEditing}}<a href="/admin/quotas">close</a>
{{else}}<a href="/admin/quotas?edit={{.Scope}}-member:{{.ID}}" title="A limit each member here has separately">per member</a>{{end}}
{{if .Editing}}<a href="/admin/quotas">{{t "quotas.close"}}</a>
{{else}}<a href="/admin/quotas?edit={{.Scope}}:{{.ID}}">{{if .HasAny}}{{t "quotas.change"}}{{else}}{{t "quotas.set"}}{{end}}</a>{{end}}
{{if .MemberEditing}}<a href="/admin/quotas">{{t "quotas.close"}}</a>
{{else}}<a href="/admin/quotas?edit={{.Scope}}-member:{{.ID}}" title="{{t "quotas.per_member_title"}}">{{t "quotas.per_member"}}</a>{{end}}
</td>
</tr>
{{if .MemberEditing}}
<tr class="editing"><td colspan="6">
{{template "quota-form" (dict "Scope" (printf "%s-member" .Scope) "ID" .ID "F" .MemberRaw "CSRF" $.CSRF
"Title" (printf "Each member of %s, separately" .Crumb)
"Note" "Every member gets this much here, counted apart from the others — a fair share rather than one pool.")}}
"Title" (t "quotas.member_form.title" .Crumb)
"Note" (t "quotas.member_form.note"))}}
</td></tr>
{{end}}
@@ -65,136 +62,121 @@ budgets bind.</p>
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}">
<input type="hidden" name="id" value="{{.ID}}">
<p class="q-form-title">Quota for <strong>{{.Crumb}}</strong>, shared by everything beneath it.
A blank field means no limit of that kind, not a limit of zero.</p>
<p class="q-form-title">{{th "quotas.form.title" .Crumb}}</p>
<div class="field-row">
<label>Requests / minute
<input type="text" name="rpm" value="{{.RPM}}" placeholder="{{if .RPMUp}}{{.RPMUp}} from {{.RPMFrom}}{{else}}none{{end}}"></label>
<label>Tokens / minute
<input type="text" name="tpm" value="{{.TPM}}" placeholder="{{if .TPMUp}}{{.TPMUp}} from {{.TPMFrom}}{{else}}none{{end}}"></label>
<label>Budget
<input type="text" name="budget" value="{{.Budget}}" placeholder="{{if .BudgetUp}}{{.BudgetUp}} from {{.BudgetFrom}}{{else}}none{{end}}"></label>
<label class="q-period">per
<label>{{t "quotas.form.rpm"}}
<input type="text" name="rpm" value="{{.RPM}}" placeholder="{{if .RPMUp}}{{t "quotas.form.up_from" .RPMUp .RPMFrom}}{{else}}{{t "common.none"}}{{end}}"></label>
<label>{{t "quotas.form.tpm"}}
<input type="text" name="tpm" value="{{.TPM}}" placeholder="{{if .TPMUp}}{{t "quotas.form.up_from" .TPMUp .TPMFrom}}{{else}}{{t "common.none"}}{{end}}"></label>
<label>{{t "quotas.col.budget"}}
<input type="text" name="budget" value="{{.Budget}}" placeholder="{{if .BudgetUp}}{{t "quotas.form.up_from" .BudgetUp .BudgetFrom}}{{else}}{{t "common.none"}}{{end}}"></label>
<label class="q-period">{{t "quotas.form.per"}}
<select name="period">
<option value="month"{{if eq .Period "month"}} selected{{end}}>month</option>
<option value="day"{{if eq .Period "day"}} selected{{end}}>day</option>
<option value="month"{{if eq .Period "month"}} selected{{end}}>{{t "quotas.per.month"}}</option>
<option value="day"{{if eq .Period "day"}} selected{{end}}>{{t "quotas.per.day"}}</option>
</select></label>
</div>
<label>Only these models <span class="opt">comma separated; blank allows all</span>
<input type="text" name="models" value="{{.Models}}" placeholder="every model"></label>
<label>{{t "quotas.form.models"}} <span class="opt">{{t "quotas.form.models_hint"}}</span>
<input type="text" name="models" value="{{.Models}}" placeholder="{{t "quotas.form.every_model"}}"></label>
<div class="q-form-actions">
<button type="submit">Save</button>
<a href="/admin/quotas">Cancel</a>
<button type="submit">{{t "common.save"}}</button>
<a href="/admin/quotas">{{t "common.cancel"}}</a>
<button type="submit" name="clear" value="1" class="link danger"
{{if not .HasAny}}disabled{{end}}>Remove every limit here</button>
{{if not .HasAny}}disabled{{end}}>{{t "quotas.form.remove_all"}}</button>
</div>
</form>
</td></tr>
{{end}}
{{end}}
</table>
<p class="q-legend"><span class="q-up">60 ↑</span> is held by a limit set higher
up — hover it to see where. A value set here is in normal type.</p>
<p class="q-legend">{{th "quotas.legend"}}</p>
{{else}}
<p class="empty">Nothing to give a quota to yet. Make an organization on the
<a href="/admin/users">Organizations &amp; people</a> tab first.</p>
<p class="empty">{{th "quotas.empty_tree"}}</p>
{{end}}
<h2 id="people-limits">People</h2>
<p class="note">A person's own limit covers everything they do, across all their
keys, projects and the chat page. {{.Personal}}</p>
<h2 id="people-limits">{{t "quotas.people.title"}}</h2>
<p class="note">{{t "quotas.people.note"}} {{.Personal}}</p>
{{if .People}}
<table class="rows quota-tree people-limits">
<tr><th>Person</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
<th class="num">Budget</th><th>Spent</th><th></th></tr>
<tr><th>{{t "quotas.col.person"}}</th><th class="num">{{t "quotas.col.rpm"}}</th><th class="num">{{t "quotas.col.tpm"}}</th>
<th class="num">{{t "quotas.col.budget"}}</th><th>{{t "quotas.col.spent"}}</th><th></th></tr>
{{range .People}}
<tr{{if .Editing}} class="editing"{{end}}>
<td class="q-where"><span class="q-name">{{.Email}}</span>
{{if .Models}}<div class="q-sub">only <code>{{.Models}}</code></div>{{end}}</td>
<td class="num">{{or .RPM "—"}}</td>
<td class="num">{{or .TPM "—"}}</td>
<td class="num">{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}{{else}}—{{end}}</td>
{{if .Models}}<div class="q-sub">{{th "quotas.only_models" .Models}}</div>{{end}}</td>
<td class="num">{{or .Shown.RPM "—"}}</td>
<td class="num">{{or .Shown.TPM "—"}}</td>
<td class="num">{{if .Budget}}{{.Shown.Budget}}{{if .Period}}<span class="q-per">/{{.Shown.Period}}</span>{{end}}{{else}}—{{end}}</td>
<td class="q-spent"><span class="q-spend">{{.Spend}}</span>
{{if .HasPct}}<span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span>
<span class="q-pct">{{.Percent}}%</span>{{end}}</td>
<td class="q-act">{{if .Editing}}<a href="/admin/quotas">close</a>
{{else}}<a href="/admin/quotas?edit=user:{{.ID}}">{{if .HasAny}}change{{else}}set{{end}}</a>{{end}}</td>
<span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}</td>
<td class="q-act">{{if .Editing}}<a href="/admin/quotas">{{t "quotas.close"}}</a>
{{else}}<a href="/admin/quotas?edit=user:{{.ID}}">{{if .HasAny}}{{t "quotas.change"}}{{else}}{{t "quotas.set"}}{{end}}</a>{{end}}</td>
</tr>
{{if .Editing}}
<tr class="editing"><td colspan="6">
{{template "quota-form" (dict "Scope" "user" "ID" .ID "CSRF" $.CSRF
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
"Title" (printf "%s, everywhere" .Email)
"Note" "Across every key this person holds, every project they use and the chat page.")}}
"Title" (t "quotas.person_form.title" .Email)
"Note" (t "quotas.person_form.note"))}}
</td></tr>
{{end}}
{{end}}
</table>
{{else}}<p class="empty">Nobody has been approved yet.</p>{{end}}
{{else}}<p class="empty">{{t "quotas.people.empty"}}</p>{{end}}
{{if .HasCapacity}}
<h2 id="capacity">Capacity</h2>
<p class="note">What a part of the tree is guaranteed of the cluster, and the most
it may hold: GPUs, CPU cores and memory, counted from the replicas of the models
it owns and of those beneath it. What it holds within its guarantee is never taken
for someone else's model; beyond it, it borrows free room and gives it back when
asked. A model's owner is set on its card on the mirror page.</p>
<h2 id="capacity">{{t "capacity.title"}}</h2>
<p class="note">{{t "capacity.note"}}</p>
{{if .Capacity}}
<table class="rows quota-tree">
<tr><th>Where</th><th class="num">GPUs</th><th class="num">Cores</th><th class="num">Memory GB</th><th>Set by</th><th></th></tr>
<tr><th>{{t "quotas.col.where"}}</th><th class="num">GPUs</th><th class="num">{{t "capacity.col.cores"}}</th><th class="num">{{t "capacity.col.memory"}}</th><th>{{t "capacity.col.set_by"}}</th><th></th></tr>
{{range .Capacity}}<tr>
<td class="q-where"><span class="q-name">{{or .Name .Scope}}</span></td>
<td class="num">{{or .GPUs "—"}}</td>
<td class="num">{{or .Cores "—"}}</td>
<td class="num">{{or .Memory "—"}}</td>
<td class="note">{{.UpdatedBy}}</td>
<td class="q-act"><a href="/admin/quotas?cap={{.Scope}}#capacity">change</a>
<td class="q-act"><a href="/admin/quotas?cap={{.Scope}}#capacity">{{t "quotas.change"}}</a>
<form method="post" action="/admin/quotas/capacity" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}">
<button type="submit" name="clear" value="1" class="link danger">remove</button>
<button type="submit" name="clear" value="1" class="link danger">{{t "capacity.remove"}}</button>
</form></td>
</tr>{{end}}
</table>
<p class="q-legend">Each figure is held of guaranteed (at most the limit); — is not governed.</p>
{{else}}<p class="empty">No capacity quotas: every model shares the cluster by its class alone.</p>{{end}}
<p class="q-legend">{{t "capacity.legend"}}</p>
{{else}}<p class="empty">{{t "capacity.empty"}}</p>{{end}}
{{if .Scopes}}
<form method="post" action="/admin/quotas/capacity" class="q-form capacity-form">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<p class="q-form-title"><strong>{{if .CapForm.Scope}}Change{{else}}Set{{end}} a capacity quota.</strong>
A blank field is not governed; 0 is none at all.</p>
<label>Where
<p class="q-form-title">{{if .CapForm.Scope}}{{th "capacity.form.title_change"}}{{else}}{{th "capacity.form.title_set"}}{{end}}</p>
<label>{{t "quotas.col.where"}}
<select name="scope" required>
<option value="">choose…</option>
<option value="">{{t "capacity.form.choose"}}</option>
{{range .Scopes}}<option value="{{.Key}}"{{if eq .Key $.CapForm.Scope}} selected{{end}}>{{.Name}}</option>{{end}}
</select></label>
<div class="field-row">
<label>Guaranteed GPUs <input type="text" name="gpus" value="{{.CapForm.GPUs}}" placeholder="—"></label>
<label>Cores <input type="text" name="cores" value="{{.CapForm.Cores}}" placeholder="—"></label>
<label>Memory GB <input type="text" name="memory" value="{{.CapForm.Memory}}" placeholder="—"></label>
<label>{{t "capacity.form.g_gpus"}} <input type="text" name="gpus" value="{{.CapForm.GPUs}}" placeholder="—"></label>
<label>{{t "capacity.col.cores"}} <input type="text" name="cores" value="{{.CapForm.Cores}}" placeholder="—"></label>
<label>{{t "capacity.col.memory"}} <input type="text" name="memory" value="{{.CapForm.Memory}}" placeholder="—"></label>
</div>
<div class="field-row">
<label>At most GPUs <input type="text" name="limit_gpus" value="{{.CapForm.LGPUs}}" placeholder="—"></label>
<label>Cores <input type="text" name="limit_cores" value="{{.CapForm.LCores}}" placeholder="—"></label>
<label>Memory GB <input type="text" name="limit_memory" value="{{.CapForm.LMem}}" placeholder="—"></label>
<label>{{t "capacity.form.l_gpus"}} <input type="text" name="limit_gpus" value="{{.CapForm.LGPUs}}" placeholder="—"></label>
<label>{{t "capacity.col.cores"}} <input type="text" name="limit_cores" value="{{.CapForm.LCores}}" placeholder="—"></label>
<label>{{t "capacity.col.memory"}} <input type="text" name="limit_memory" value="{{.CapForm.LMem}}" placeholder="—"></label>
</div>
<div class="q-form-actions"><button type="submit">Save</button>
{{if .CapForm.Scope}}<a href="/admin/quotas#capacity">Cancel</a>{{end}}</div>
<div class="q-form-actions"><button type="submit">{{t "common.save"}}</button>
{{if .CapForm.Scope}}<a href="/admin/quotas#capacity">{{t "common.cancel"}}</a>{{end}}</div>
</form>
{{else}}<p class="empty">Nothing to give a capacity quota to yet. Make an organization on the
<a href="/admin/users">Organizations &amp; people</a> tab first.</p>{{end}}
{{else}}<p class="empty">{{th "capacity.empty_tree"}}</p>{{end}}
{{end}}
<details class="explain">
<summary>How limits combine</summary>
<p>Along a branch the most restrictive value wins, so a team cannot raise what
its organization set and a project cannot raise its team's. Only an
organization's own limit is bounded by nothing. Model lists intersect: a
project cannot allow a model its team withheld.</p>
<p>Per-key limits are not set here: a key is bounded by its project, and
putting every caller's throughput one click from an admin session is more
than this page should do — <code>llmgw keys limits --scope key</code>.</p>
<summary>{{t "quotas.explain.summary"}}</summary>
<p>{{t "quotas.explain.chain"}}</p>
<p>{{th "quotas.explain.keys"}}</p>
</details>
{{end}}
@@ -209,26 +191,26 @@ asked. A model's owner is set on its card on the mirror page.</p>
<input type="hidden" name="id" value="{{.ID}}">
{{/* From a panel on the people tab: the answer comes back to that panel. */}}
{{with .Sel}}<input type="hidden" name="tab" value="people"><input type="hidden" name="sel" value="{{.}}">{{end}}
<p class="q-form-title"><strong>{{.Title}}</strong>. {{.Note}} A blank field means no limit of that kind.</p>
<p class="q-form-title">{{th "quotas.qform.title" .Title .Note}}</p>
<div class="field-row">
<label>Requests / minute <input type="text" name="rpm" value="{{.F.RPM}}" placeholder="none"></label>
<label>Tokens / minute <input type="text" name="tpm" value="{{.F.TPM}}" placeholder="none"></label>
<label>Budget <input type="text" name="budget" value="{{.F.Budget}}" placeholder="none"></label>
<label class="q-period">per
<label>{{t "quotas.form.rpm"}} <input type="text" name="rpm" value="{{.F.RPM}}" placeholder="{{t "common.none"}}"></label>
<label>{{t "quotas.form.tpm"}} <input type="text" name="tpm" value="{{.F.TPM}}" placeholder="{{t "common.none"}}"></label>
<label>{{t "quotas.col.budget"}} <input type="text" name="budget" value="{{.F.Budget}}" placeholder="{{t "common.none"}}"></label>
<label class="q-period">{{t "quotas.form.per"}}
<select name="period">
<option value="month"{{if eq .F.Period "month"}} selected{{end}}>month</option>
<option value="day"{{if eq .F.Period "day"}} selected{{end}}>day</option>
<option value="month"{{if eq .F.Period "month"}} selected{{end}}>{{t "quotas.per.month"}}</option>
<option value="day"{{if eq .F.Period "day"}} selected{{end}}>{{t "quotas.per.day"}}</option>
</select></label>
</div>
<label>Only these models <span class="opt">comma separated; blank allows all</span>
<input type="text" name="models" value="{{.F.Models}}" placeholder="every model"></label>
<label>{{t "quotas.form.models"}} <span class="opt">{{t "quotas.form.models_hint"}}</span>
<input type="text" name="models" value="{{.F.Models}}" placeholder="{{t "quotas.form.every_model"}}"></label>
<div class="q-form-actions">
<button type="submit">Save</button>
{{if not .Sel}}<a href="/admin/quotas">Cancel</a>{{end}}
<button type="submit">{{t "common.save"}}</button>
{{if not .Sel}}<a href="/admin/quotas">{{t "common.cancel"}}</a>{{end}}
<button type="submit" name="clear" value="1" class="link danger"
{{if not .F.HasAny}}disabled{{end}}>Remove these limits</button>
{{if not .F.HasAny}}disabled{{end}}>{{t "quotas.qform.remove"}}</button>
</div>
</form>
{{end}}
{{define "quota-value"}}{{if .Own}}{{.Own}}{{else if .Up}}<span class="q-up" title="set on {{.From}}">{{.Up}} ↑</span>{{else}}<span class="q-none">—</span>{{end}}{{end}}
{{define "quota-value"}}{{if .Own}}{{.Own}}{{else if .Up}}<span class="q-up" title="{{t "quotas.set_on" .From}}">{{.Up}} ↑</span>{{else}}<span class="q-none">—</span>{{end}}{{end}}
+25 -32
View File
@@ -7,53 +7,51 @@
them. Links rather than a script: the choice is in the URL, so a view worth
coming back to can be bookmarked or pasted to somebody else. */}}
<div class="filters">
<span class="filter-label">Group by</span>
<span class="filter-label">{{t "usage.group_by"}}</span>
{{range .Groups}}<a class="chip{{if .Selected}} on{{end}}"
href="/admin/usage?by={{.Key}}&amp;window={{$.Window}}">{{.Label}}</a>{{end}}
<span class="filter-label">Window</span>
<span class="filter-label">{{t "usage.window"}}</span>
{{range .Windows}}<a class="chip{{if .Selected}} on{{end}}"
href="/admin/usage?by={{$.By}}&amp;window={{.Key}}">{{.Label}}</a>{{end}}
</div>
<div class="chart-row">
{{template "tschart" dict "t" .Requests "title" "Requests"
"note" "Requests served in each slice, counted where the key belongs now rather than where it was at the time."}}
{{template "tschart" dict "t" .Tokens "title" "Tokens"
"note" "Prompt and completion together, which is the figure a person recognises as what they consumed."}}
{{template "tschart" dict "t" .Requests "title" (t "usage.requests")
"note" (t "usage.requests_note")}}
{{template "tschart" dict "t" .Tokens "title" (t "usage.tokens")
"note" (t "usage.tokens_note")}}
</div>
<div class="chart-row">
{{if .Priced}}
{{template "tschart" dict "t" .Spend "title" "Spend"
"note" "In whole units, the same ones a budget is written in."}}
{{template "tschart" dict "t" .Spend "title" (t "usage.spend")
"note" (t "usage.spend_note")}}
{{else}}
{{/* Said rather than drawn. A spend chart flat along the bottom reads as a
quiet month, and this is not that. */}}
<div class="chart-wrap">
<h3 class="chart-title">Spend</h3>
<p class="empty">No model has a price, so every request costs zero and this
chart would be a flat line however busy the cluster is. Set prices in the
<code>pricing</code> block of the server configuration.</p>
<h3 class="chart-title">{{t "usage.spend"}}</h3>
<p class="empty">{{th "usage.unpriced"}}</p>
</div>
{{end}}
</div>
{{if .Rows}}
<table class="rows">
<tr><th>Group</th><th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th>
{{if .HasStats}}<th class="num" title="Failed requests, refusals by a limit included">Error rate</th>
<th class="num" title="Refused by a rate, token or budget limit">Limited</th>
<th class="num" title="Time to first token, for streamed requests">First token p50 / p95</th>
<th class="num">Latency p95</th>{{end}}</tr>
<tr><th>{{t "usage.col.group"}}</th><th class="num">{{t "usage.requests"}}</th><th class="num">{{t "usage.tokens"}}</th><th class="num">{{t "usage.spend"}}</th><th class="num">{{t "usage.errors"}}</th>
{{if .HasStats}}<th class="num" title="{{t "usage.col.error_rate_title"}}">{{t "usage.col.error_rate"}}</th>
<th class="num" title="{{t "usage.col.limited_title"}}">{{t "usage.col.limited"}}</th>
<th class="num" title="{{t "usage.col.ttft_title"}}">{{t "usage.col.ttft"}}</th>
<th class="num">{{t "usage.col.latency"}}</th>{{end}}</tr>
{{range .Rows}}
<tr>
<td>{{.Name}}</td>
<td class="num">{{compact .Requests}}</td>
<td class="num">{{compact .Tokens}}</td>
<td class="num">{{.Spend}}</td>
<td class="num">{{if .Errors}}{{.Errors}}{{else}}—{{end}}</td>
<td class="num">{{if .Errors}}{{count .Errors}}{{else}}—{{end}}</td>
{{if $.HasStats}}{{with .Stats}}
<td class="num{{if ge .ErrorRate 5.0}} q-warn{{end}}">{{if .Requests}}{{pct1 .ErrorRate}}%{{else}}—{{end}}</td>
<td class="num{{if ge .ErrorRate 5.0}} q-warn{{end}}">{{if .Requests}}{{t "usage.pct" (pct1 .ErrorRate)}}{{else}}—{{end}}</td>
<td class="num{{if .Limited}} q-warn{{end}}">{{if .Limited}}{{compact .Limited}}{{else}}—{{end}}</td>
<td class="num">{{pctile .TTFT 50}} / {{pctile .TTFT 95}}</td>
<td class="num">{{pctile .Latency 95}}</td>{{end}}{{end}}
@@ -61,34 +59,29 @@
{{end}}
</table>
{{else}}
<p class="empty">Nothing recorded in this window.</p>
<p class="empty">{{t "usage.empty"}}</p>
{{end}}
{{with .GPUHours}}
<h2>GPU time, by owner</h2>
<h2>{{t "usage.gpu.title"}}</h2>
{{/* What each owner's models held, whether or not anybody asked them
anything: a model kept loaded holds its cards all the same, and that is
the time a GPU quota is about. */}}
<div class="chart-row">
{{template "tschart" dict "t" $.GPUHeld "title" "GPUs held, by owner" "note" "The cards held by the replicas of each owner's models: models whose owner is a part of the quota tree."}}
{{template "tschart" dict "t" $.GPUHeld "title" (t "usage.gpu.chart") "note" (t "usage.gpu.chart_note")}}
</div>
<table class="rows">
<tr><th>Owner</th><th class="num">GPU-hours</th><th class="num">Held now</th></tr>
{{range .}}<tr><td>{{.Owner}}</td><td class="num">{{printf "%.1f" .Hours}}</td><td class="num">{{printf "%.0f" .Now}}</td></tr>{{end}}
<tr><th>{{t "usage.col.owner"}}</th><th class="num">{{t "usage.col.gpu_hours"}}</th><th class="num">{{t "usage.col.held_now"}}</th></tr>
{{range .}}<tr><td>{{.Owner}}</td><td class="num">{{num .Hours 1}}</td><td class="num">{{num .Now 0}}</td></tr>{{end}}
</table>
<p class="note">From the metrics store, which keeps its history for as long as it is set to (fifteen days by default); a model with no owner is counted under “no owner”.</p>
<p class="note">{{t "usage.gpu.note"}}</p>
{{end}}
{{/* The charts fold their tail; the table does not. Both are here because they
answer different questions, and the one that is easy to get wrong is
thinking the four coloured lines are all of it. */}}
<p class="note">The charts draw the four busiest groups by total and sum the
rest into “other”, so they always add up to the whole cluster. The table below
is every group, in the same order.</p>
<p class="note">{{t "usage.note.charts"}}</p>
<p class="note">A person is not a level of the quota tree — one person holds
keys in several projects — so grouping by person crosses the tree rather than
naming a level of it. Quotas are written on the other three, on the
<a href="/admin/quotas">Limits</a> tab.</p>
<p class="note">{{th "usage.note.person"}}</p>
{{end}}
+84 -93
View File
@@ -5,59 +5,55 @@
{{if .Message}}<p class="ok" role="status">{{.Message}}</p>{{end}}
{{if .Pending}}
<h2>Waiting for approval</h2>
<table class="rows"><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Approve, and put them in</th><th></th></tr>
<h2>{{t "users.pending.title"}}</h2>
<table class="rows"><tr><th>{{t "users.col.email"}}</th><th>{{t "users.col.name"}}</th><th>{{t "users.col.signed_up"}}</th><th>{{t "users.col.approve_into"}}</th><th></th></tr>
{{range .Pending}}<tr>
<td>{{.Email}}</td><td>{{.Name}}</td><td>{{ago .CreatedAt}}</td>
<td>
<form method="post" action="/admin/users/{{.ID}}/approve" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<select name="project" aria-label="Project for {{.Email}}">
<option value="">No project yet: on their own</option>
<select name="project" aria-label="{{t "users.project_for" .Email}}">
<option value="">{{t "users.no_project"}}</option>
{{range $.Projects}}<option value="{{.ID}}">{{.Path}}</option>{{end}}
</select>
<button type="submit">Approve</button>
<button type="submit">{{t "users.approve"}}</button>
</form>
</td>
<td>
<form method="post" action="/admin/users/{{.ID}}/delete" class="inline"
hx-confirm="Reject and remove {{.Email}}?">
hx-confirm="{{t "users.reject_confirm" .Email}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">Reject</button>
<button type="submit" class="link danger">{{t "users.reject"}}</button>
</form>
</td>
</tr>{{end}}</table>
{{end}}
<h2>Organizations, teams and projects</h2>
<p class="note">A person is in a project, a project in a team, a team in an
organization{{if .HasQuotas}}, and <a href="/admin/quotas">limits</a> can be set
on any of them{{end}}. Choose a name — or a person in the list below — to see
its limits, capacity and use beside the tree; open a project for its people and keys.</p>
<h2>{{t "users.tree.title"}}</h2>
<p class="note">{{if .HasQuotas}}{{th "users.tree.note_quotas"}}{{else}}{{t "users.tree.note"}}{{end}}</p>
{{if not .Tree}}
<p class="empty">No organizations yet. Create one, then a team in it, then a
project in the team — an account can only be approved into a project.</p>
<p class="empty">{{t "users.empty"}}</p>
{{end}}
{{range $org := .Tree}}
<section class="org">
<div class="node-head">
<strong class="node-name"><a href="/admin/users?sel=org:{{$org.ID}}" class="sel-link{{if eq $.Sel (printf "org:%s" $org.ID)}} on{{end}}">{{$org.Name}}</a></strong>
<span class="node-meta">{{len $org.Teams}} team{{if ne (len $org.Teams) 1}}s{{end}}</span>
<span class="node-meta">{{tn "users.n_teams" (len $org.Teams)}}</span>
{{template "scope-members" (dict "Kind" "org" "ID" $org.ID "Name" $org.Name "D" $)}}
<details class="edit"><summary>rename</summary>
<details class="edit"><summary>{{t "users.rename"}}</summary>
<form method="post" action="/admin/tenancy/org/{{$org.ID}}/rename">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="text" name="name" value="{{$org.Name}}" required aria-label="New name">
<button type="submit">Save</button>
<input type="text" name="name" value="{{$org.Name}}" required aria-label="{{t "users.new_name"}}">
<button type="submit">{{t "common.save"}}</button>
</form>
</details>
{{if not $org.Teams}}
<form method="post" action="/admin/tenancy/org/{{$org.ID}}/delete" class="inline"
hx-confirm="Remove the organization {{$org.Name}}?">
hx-confirm="{{t "users.remove_org_confirm" $org.Name}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">remove</button>
<button type="submit" class="link danger">{{t "users.remove"}}</button>
</form>
{{end}}
</div>
@@ -66,31 +62,31 @@ project in the team — an account can only be approved into a project.</p>
<div class="team">
<div class="node-head">
<span class="node-name"><a href="/admin/users?sel=team:{{$team.ID}}" class="sel-link{{if eq $.Sel (printf "team:%s" $team.ID)}} on{{end}}">{{$team.Name}}</a></span>
<span class="node-meta">{{len $team.Projects}} project{{if ne (len $team.Projects) 1}}s{{end}}</span>
<span class="node-meta">{{tn "users.n_projects" (len $team.Projects)}}</span>
{{template "scope-members" (dict "Kind" "team" "ID" $team.ID "Name" $team.Name "D" $)}}
<details class="edit"><summary>rename</summary>
<details class="edit"><summary>{{t "users.rename"}}</summary>
<form method="post" action="/admin/tenancy/team/{{$team.ID}}/rename">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="text" name="name" value="{{$team.Name}}" required aria-label="New name">
<button type="submit">Save</button>
<input type="text" name="name" value="{{$team.Name}}" required aria-label="{{t "users.new_name"}}">
<button type="submit">{{t "common.save"}}</button>
</form>
</details>
{{if gt (len $.Orgs) 1}}
<details class="edit"><summary>move</summary>
<details class="edit"><summary>{{t "users.move"}}</summary>
<form method="post" action="/admin/tenancy/team/{{$team.ID}}/move">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<select name="to" aria-label="Organization to move {{$team.Name}} to">
<select name="to" aria-label="{{t "users.move_team_to" $team.Name}}">
{{range $.Orgs}}{{if ne .ID $org.ID}}<option value="{{.ID}}">{{.Name}}</option>{{end}}{{end}}
</select>
<button type="submit">Move</button>
<button type="submit">{{t "users.move_button"}}</button>
</form>
</details>
{{end}}
{{if not $team.Projects}}
<form method="post" action="/admin/tenancy/team/{{$team.ID}}/delete" class="inline"
hx-confirm="Remove the team {{$team.Name}}?">
hx-confirm="{{t "users.remove_team_confirm" $team.Name}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">remove</button>
<button type="submit" class="link danger">{{t "users.remove"}}</button>
</form>
{{end}}
</div>
@@ -99,61 +95,61 @@ project in the team — an account can only be approved into a project.</p>
<details class="project" id="p-{{$p.ID}}"{{if eq $.Open $p.ID}} open{{end}}>
<summary>
<span class="node-name">{{$p.Name}}</span>
<a href="/admin/users?sel=project:{{$p.ID}}&amp;open={{$p.ID}}" class="sel-link small{{if eq $.Sel (printf "project:%s" $p.ID)}} on{{end}}" title="Limits and use of {{$p.Name}}">limits &amp; use</a>
<span class="node-meta">{{$p.Users}} member{{if ne $p.Users 1}}s{{end}}
· {{$p.ActiveKeys}} key{{if ne $p.ActiveKeys 1}}s{{end}}{{if gt $p.Keys $p.ActiveKeys}} ({{sub $p.Keys $p.ActiveKeys}} revoked){{end}}</span>
<a href="/admin/users?sel=project:{{$p.ID}}&amp;open={{$p.ID}}" class="sel-link small{{if eq $.Sel (printf "project:%s" $p.ID)}} on{{end}}" title="{{t "users.limits_of" $p.Name}}">{{t "users.limits_and_use"}}</a>
<span class="node-meta">{{tn "users.n_members" $p.Users}}
· {{tn "users.n_keys" $p.ActiveKeys}}{{if gt $p.Keys $p.ActiveKeys}} {{t "users.n_revoked" (sub $p.Keys $p.ActiveKeys)}}{{end}}</span>
</summary>
<div class="project-body">
<div class="node-head">
<details class="edit"><summary>rename</summary>
<details class="edit"><summary>{{t "users.rename"}}</summary>
<form method="post" action="/admin/tenancy/project/{{$p.ID}}/rename">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="open" value="{{$p.ID}}">
<input type="text" name="name" value="{{$p.Name}}" required aria-label="New name">
<button type="submit">Save</button>
<input type="text" name="name" value="{{$p.Name}}" required aria-label="{{t "users.new_name"}}">
<button type="submit">{{t "common.save"}}</button>
</form>
</details>
{{if gt (len $.Teams) 1}}
<details class="edit"><summary>move to another team</summary>
<details class="edit"><summary>{{t "users.move_project"}}</summary>
<form method="post" action="/admin/tenancy/project/{{$p.ID}}/move">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="open" value="{{$p.ID}}">
<select name="to" aria-label="Team to move {{$p.Name}} to">
<select name="to" aria-label="{{t "users.move_project_to" $p.Name}}">
{{range $.Teams}}{{if ne .ID $team.ID}}<option value="{{.ID}}">{{.OrgName}} / {{.Name}}</option>{{end}}{{end}}
</select>
<button type="submit">Move</button>
<button type="submit">{{t "users.move_button"}}</button>
</form>
</details>
{{end}}
{{if and (eq $p.Users 0) (eq $p.Keys 0)}}
<form method="post" action="/admin/tenancy/project/{{$p.ID}}/delete" class="inline"
hx-confirm="Remove the project {{$p.Name}}?">
hx-confirm="{{t "users.remove_project_confirm" $p.Name}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">remove</button>
<button type="submit" class="link danger">{{t "users.remove"}}</button>
</form>
{{else}}<span class="note">Removable once it has no people and never had a key.</span>{{end}}
{{else}}<span class="note">{{t "users.not_removable"}}</span>{{end}}
</div>
{{with index $.Members (printf "project:%s" $p.ID)}}
<table class="rows compact"><tr><th>Member</th><th>State</th><th>Last sign-in</th><th></th></tr>
<table class="rows compact"><tr><th>{{t "users.col.member"}}</th><th>{{t "users.col.state"}}</th><th>{{t "users.col.last_sign_in"}}</th><th></th></tr>
{{range .}}{{template "member-row" (dict "U" . "D" $ "Scope" (printf "project:%s" $p.ID) "Open" $p.ID)}}{{end}}
</table>
{{else}}<p class="empty">No members of its own. Members of its team or organization can use it too.</p>{{end}}
{{else}}<p class="empty">{{t "users.no_members"}}</p>{{end}}
{{template "add-member" (dict "Kind" "project" "ID" $p.ID "D" $ "Open" $p.ID)}}
{{with index $.Keys $p.ID}}
<table class="rows compact"><tr><th>Key</th><th>Owner</th><th>State</th><th>Last used</th><th></th></tr>
<table class="rows compact"><tr><th>{{t "users.col.key"}}</th><th>{{t "users.col.owner"}}</th><th>{{t "users.col.state"}}</th><th>{{t "users.col.last_used"}}</th><th></th></tr>
{{range .}}<tr>
<td>{{.Name}} <code>{{.Prefix}}</code></td>
<td>{{or .OwnerEmail "issued by an operator"}}</td>
<td class="state state-{{.State}}">{{.State}}</td>
<td>{{or .OwnerEmail (t "users.issued_by_operator")}}</td>
<td class="state state-{{.State}}">{{t (print "users.key_state." .State)}}</td>
<td>{{ago .LastUsedAt}}</td>
<td>{{if eq .State "active"}}
<form method="post" action="/admin/keys/{{.ID}}/revoke" class="inline"
hx-confirm="Revoke the key {{.Name}}? Whatever uses it stops working.">
hx-confirm="{{t "users.revoke_confirm" .Name}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="open" value="{{$p.ID}}">
<button type="submit" class="link danger">revoke</button>
<button type="submit" class="link danger">{{t "users.revoke"}}</button>
</form>{{end}}</td>
</tr>{{end}}
</table>
@@ -165,24 +161,24 @@ project in the team — an account can only be approved into a project.</p>
{{/* Folded until wanted: a box open under every team was most of the
tree's height and read as something to fill in. */}}
<details class="add-inline">
<summary>+ project</summary>
<summary>{{t "users.add_project"}}</summary>
<form method="post" action="/admin/tenancy/project" class="add">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="team" value="{{$team.ID}}">
<input type="text" name="name" required placeholder="new project in {{$team.Name}}" aria-label="New project in {{$team.Name}}">
<button type="submit">Add</button>
<input type="text" name="name" required placeholder="{{t "users.new_project_in" $team.Name}}" aria-label="{{t "users.new_project_in_label" $team.Name}}">
<button type="submit">{{t "common.add"}}</button>
</form>
</details>
</div>
{{end}}
<details class="add-inline team-add">
<summary>+ team</summary>
<summary>{{t "users.add_team"}}</summary>
<form method="post" action="/admin/tenancy/team" class="add">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="org" value="{{$org.ID}}">
<input type="text" name="name" required placeholder="new team in {{$org.Name}}" aria-label="New team in {{$org.Name}}">
<button type="submit">Add</button>
<input type="text" name="name" required placeholder="{{t "users.new_team_in" $org.Name}}" aria-label="{{t "users.new_team_in_label" $org.Name}}">
<button type="submit">{{t "common.add"}}</button>
</form>
</details>
</section>
@@ -190,24 +186,19 @@ project in the team — an account can only be approved into a project.</p>
<form method="post" action="/admin/tenancy/org" class="add org-add">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<input type="text" name="name" required placeholder="new organization" aria-label="New organization">
<button type="submit">Add organization</button>
<input type="text" name="name" required placeholder="{{t "users.new_org"}}" aria-label="{{t "users.new_org_label"}}">
<button type="submit">{{t "users.add_org"}}</button>
</form>
<h2 id="accounts">All accounts</h2>
<h2 id="accounts">{{t "users.accounts"}}</h2>
{{if .Accounts}}
<table class="rows" id="account-list"><tr><th>Person</th><th>State</th><th>Member of</th><th>Last sign-in</th><th>Add to</th><th></th></tr>
<table class="rows" id="account-list"><tr><th>{{t "users.col.person"}}</th><th>{{t "users.col.state"}}</th><th>{{t "users.col.member_of"}}</th><th>{{t "users.col.last_sign_in"}}</th><th>{{t "users.col.add_to"}}</th><th></th></tr>
{{range .Accounts}}{{template "account-row" (dict "U" . "D" $)}}{{end}}
</table>
<p class="note">Someone who is a member of nothing uses the gateway on their own:
personal keys and the chat page, within their personal limits.</p>
{{else}}<p class="empty">No accounts yet.</p>{{end}}
<p class="note">{{t "users.alone_note"}}</p>
{{else}}<p class="empty">{{t "users.no_accounts"}}</p>{{end}}
<p class="note">Not here on purpose: making someone an admin, resetting a password
and issuing a key for someone else. Each turns a stolen admin session into more
access than it had, so they need a shell on the control plane —
<code>llmgw users promote</code>, <code>llmgw users reset-password</code>,
<code>llmgw keys create</code>.</p>
<p class="note">{{th "users.shell_note"}}</p>
{{end}}
@@ -217,56 +208,56 @@ access than it had, so they need a shell on the control plane —
{{/* In a menu: two red links on every row made the list look alarmed,
and they are the two things done least often. */}}
<details class="row-menu">
<summary aria-label="More for {{$u.Email}}" title="More">⋯</summary>
<summary aria-label="{{t "users.more_for" $u.Email}}" title="{{t "users.more"}}">⋯</summary>
<div class="menu">
{{if eq (printf "%s" $u.State) "active"}}
<form method="post" action="/admin/users/{{$u.ID}}/suspend" class="inline">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="open" value="{{.Open}}">
<button type="submit" class="link danger">suspend</button>
<button type="submit" class="link danger">{{t "users.suspend"}}</button>
</form>
{{else}}
<form method="post" action="/admin/users/{{$u.ID}}/reinstate" class="inline">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="open" value="{{.Open}}">
<button type="submit" class="link">reinstate</button>
<button type="submit" class="link">{{t "users.reinstate"}}</button>
</form>
{{end}}
<form method="post" action="/admin/users/{{$u.ID}}/delete" class="inline"
hx-confirm="Remove {{$u.Email}} for good? Their keys are revoked.">
hx-confirm="{{t "users.remove_confirm" $u.Email}}">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="open" value="{{.Open}}">
<button type="submit" class="link danger">remove</button>
<button type="submit" class="link danger">{{t "users.remove"}}</button>
</form>
</div>
</details>
{{else}}<span class="note">you</span>{{end}}
{{else}}<span class="note">{{t "users.you"}}</span>{{end}}
{{end}}
{{/* A person in the list of every account: where they are, and where else
they could be. */}}
{{define "account-row"}}{{$u := .U}}{{$d := .D}}
<tr data-q="{{lower $u.Email}} {{lower $u.Name}}{{range $u.Memberships}} {{lower .Path}}{{end}}">
<td><a href="/admin/users?sel=user:{{$u.ID}}" class="sel-link{{if eq $d.Sel (printf "user:%s" $u.ID)}} on{{end}}">{{$u.Email}}</a>{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
<td class="state state-{{$u.State}}">{{$u.State}}</td>
<td><a href="/admin/users?sel=user:{{$u.ID}}" class="sel-link{{if eq $d.Sel (printf "user:%s" $u.ID)}} on{{end}}">{{$u.Email}}</a>{{if $u.IsAdmin}} <span class="tag">{{t "users.tag.admin"}}</span>{{end}}</td>
<td class="state state-{{$u.State}}">{{t (print "users.state." $u.State)}}</td>
<td class="where">
{{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
{{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">{{t "users.role.manager"}}</b>{{end}}
{{if eq (printf "%s" .Scope) "org"}}
<form method="post" action="/admin/users/{{$u.ID}}/role" class="inline">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="scope" value="org:{{.ScopeID}}">
<input type="hidden" name="role" value="{{if eq .Role "manager"}}member{{else}}manager{{end}}">
<button type="submit" class="chip-x" title="{{if eq .Role "manager"}}Stop managing {{.Path}}{{else}}Make manager of {{.Path}}: can see its page{{end}}"
aria-label="{{if eq .Role "manager"}}Stop managing{{else}}Make manager{{end}}">{{if eq .Role "manager"}}☆{{else}}★{{end}}</button>
<button type="submit" class="chip-x" title="{{if eq .Role "manager"}}{{t "users.stop_managing_of" .Path}}{{else}}{{t "users.make_manager_of" .Path}}{{end}}"
aria-label="{{if eq .Role "manager"}}{{t "users.stop_managing"}}{{else}}{{t "users.make_manager"}}{{end}}">{{if eq .Role "manager"}}☆{{else}}★{{end}}</button>
</form>
{{end}}
<form method="post" action="/admin/users/{{$u.ID}}/leave" class="inline"
hx-confirm="Take {{$u.Email}} out of {{.Path}}? Keys they made there are revoked.">
hx-confirm="{{t "users.take_out_confirm" $u.Email .Path}}">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}:{{.ScopeID}}">
<button type="submit" class="chip-x" aria-label="Take out of {{.Path}}" title="Take out of {{.Path}}">×</button>
<button type="submit" class="chip-x" aria-label="{{t "users.take_out_of" .Path}}" title="{{t "users.take_out_of" .Path}}">×</button>
</form></span>{{end}}{{end}}
{{else}}<span class="opt">on their own</span>{{end}}
{{else}}<span class="opt">{{t "users.on_own"}}</span>{{end}}
{{/* Two, then a count: a person in six teams was six chips wrapping the
row into a paragraph. The rest are named on hover and in their own
scopes' member lists, where taking them out is done. */}}
@@ -276,11 +267,11 @@ access than it had, so they need a shell on the control plane —
<td>
<form method="post" action="/admin/users/{{$u.ID}}/join" class="inline add-to">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<select name="scope" aria-label="Add {{$u.Email}} to" required>
<option value="">choose…</option>
<select name="scope" aria-label="{{t "users.add_to_label" $u.Email}}" required>
<option value="">{{t "users.choose"}}</option>
{{range $d.Scopes}}<option value="{{.Value}}">{{.Label}}</option>{{end}}
</select>
<button type="submit" class="small">Add</button>
<button type="submit" class="small">{{t "common.add"}}</button>
</form>
</td>
<td class="actions">{{template "state-actions" (dict "U" $u "D" $d "Open" "")}}</td>
@@ -289,17 +280,17 @@ access than it had, so they need a shell on the control plane —
{{/* A member in a scope's own list: taking them out of here is the action. */}}
{{define "member-row"}}{{$u := .U}}{{$d := .D}}
<tr>
<td>{{$u.Email}}{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}
{{if gt (len $u.Memberships) 1}}<span class="opt">also in {{sub (len $u.Memberships) 1}} more</span>{{end}}</td>
<td class="state state-{{$u.State}}">{{$u.State}}</td>
<td>{{$u.Email}}{{if $u.IsAdmin}} <span class="tag">{{t "users.tag.admin"}}</span>{{end}}
{{if gt (len $u.Memberships) 1}}<span class="opt">{{t "users.also_in" (sub (len $u.Memberships) 1)}}</span>{{end}}</td>
<td class="state state-{{$u.State}}">{{t (print "users.state." $u.State)}}</td>
<td>{{ago $u.LastLoginAt}}</td>
<td class="actions">
<form method="post" action="/admin/users/{{$u.ID}}/leave" class="inline"
hx-confirm="Take {{$u.Email}} out of here? Keys they made here are revoked.">
hx-confirm="{{t "users.take_out_here_confirm" $u.Email}}">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}">
<input type="hidden" name="open" value="{{.Open}}">
<button type="submit" class="link danger">take out</button>
<button type="submit" class="link danger">{{t "users.take_out"}}</button>
</form>
</td>
</tr>{{end}}
@@ -309,22 +300,22 @@ access than it had, so they need a shell on the control plane —
<form method="post" action="/admin/tenancy/{{.Kind}}/{{.ID}}/add" class="add">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="open" value="{{.Open}}">
<select name="user" aria-label="Person to add" required>
<option value="">choose a person…</option>
<select name="user" aria-label="{{t "users.person_to_add"}}" required>
<option value="">{{t "users.choose_person"}}</option>
{{range $d.Accounts}}<option value="{{.ID}}">{{.Email}}</option>{{end}}
</select>
<button type="submit">Add member</button>
<button type="submit">{{t "users.add_member"}}</button>
</form>{{end}}
{{/* An org's or team's own members, folded away until asked for. */}}
{{define "scope-members"}}{{$d := .D}}{{$k := printf "%s:%s" .Kind .ID}}
<details class="edit"><summary>{{len (index $d.Members $k)}} member{{if ne (len (index $d.Members $k)) 1}}s{{end}}</summary>
<details class="edit"><summary>{{tn "users.n_members" (len (index $d.Members $k))}}</summary>
<div class="scope-members">
{{with index $d.Members $k}}
<table class="rows compact">
{{range .}}{{template "member-row" (dict "U" . "D" $d "Scope" $k "Open" "")}}{{end}}
</table>
{{else}}<p class="note">Nobody is a member of {{.Name}} itself; a member here could use every project in it.</p>{{end}}
{{else}}<p class="note">{{t "users.nobody_in" .Name}}</p>{{end}}
{{template "add-member" (dict "Kind" .Kind "ID" .ID "D" $d "Open" "")}}
</div>
</details>{{end}}
+82 -59
View File
@@ -1,7 +1,7 @@
{{define "chat"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Chat · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "chat.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<link rel="stylesheet" href="/_static/katex/katex.min.css">
<script defer src="/_static/katex/katex.min.js"></script>
<script defer src="/_static/katex/auto-render.min.js"></script>
@@ -9,7 +9,7 @@
<meta name="csrf" content="{{.CSRF}}">
</head><body><div class="wrap">
<header>
<h1>Chat</h1>
<h1>{{t "chat.title"}}</h1>
<p class="sub">{{.User.Email}}</p>
{{template "nav" .Nav}}
</header>
@@ -19,62 +19,85 @@
below is the one this choice allows; the server checks it again on every
message rather than trusting the page. */}}
<form method="get" action="/chat" class="chat-context">
<label class="inline-label">Using
<label class="inline-label">{{t "chat.context.using"}}
<select id="context" name="context" data-autosubmit>
{{range .Contexts}}<option value="{{.Value}}"{{if eq .Value $.Context}} selected{{end}}>{{.Label}}</option>{{end}}
</select></label>
<noscript><button type="submit" class="link">switch</button></noscript>
{{/* translate="no" only because a parser reads noscript as raw text and
would take the markup for prose; the label is translated. */}}
<noscript translate="no"><button type="submit" class="link">{{t "chat.context.switch"}}</button></noscript>
</form>
{{else if .Contexts}}{{with index .Contexts 0}}<input type="hidden" id="context" value="{{.Value}}">{{end}}{{end}}
{{if .Models}}
<div class="chat-bar">
<label class="inline-label">Model
<select id="model">{{range .Models}}<option value="{{.Name}}" data-simulated="{{if .Simulated}}1{{else}}0{{end}}" data-ondemand="{{if .OnDemand}}1{{else}}0{{end}}">{{.Name}}{{with .Runs}} · {{.}}{{end}}{{if .Simulated}} (simulated){{end}}{{if .OnDemand}} (not loaded){{end}}</option>{{end}}</select>
<label class="inline-label">{{t "chat.model"}}
<select id="model">{{range .Models}}<option value="{{.Name}}" data-simulated="{{if .Simulated}}1{{else}}0{{end}}" data-ondemand="{{if .OnDemand}}1{{else}}0{{end}}">{{.Name}}{{with .Runs}} · {{.}}{{end}}{{if .Simulated}} {{t "chat.model.simulated"}}{{end}}{{if .OnDemand}} {{t "chat.model.not_loaded"}}{{end}}</option>{{end}}</select>
</label>
<button id="reset" type="button" class="link">New conversation</button>
<button id="reset" type="button" class="link">{{t "chat.new"}}</button>
</div>
<details class="chat-settings" id="settings">
<summary>Settings</summary>
<label>System prompt <span class="opt">how the model should behave, sent before every turn</span>
<textarea id="system" rows="2" placeholder="You are a concise assistant. Answer in the language of the question."></textarea></label>
<label class="temp">Temperature <output id="temp-out">0.7</output>
<summary>{{t "chat.settings"}}</summary>
<label>{{t "chat.settings.system"}} <span class="opt">{{t "chat.settings.system_hint"}}</span>
<textarea id="system" rows="2" placeholder="{{t "chat.settings.system_placeholder"}}"></textarea></label>
<label class="temp">{{t "chat.settings.temperature"}} <output id="temp-out">{{num 0.7 1}}</output>
<input type="range" id="temperature" min="0" max="2" step="0.1" value="0.7">
<span class="opt">lower is steadier, higher more varied</span></label>
<span class="opt">{{t "chat.settings.temperature_hint"}}</span></label>
</details>
<p id="sim-note" class="note" hidden>
This model is simulated: it exercises scheduling and reports timings, and its
replies are placeholder text, not inference.
</p>
<p id="sim-note" class="note" hidden>{{t "chat.note.simulated"}}</p>
<p id="ondemand-note" class="note" hidden>
This model is not running. The first message loads it onto a node — fetching
its weights and starting the engine, minutes for a large model — and is sent
once it is up.
</p>
<p id="ondemand-note" class="note" hidden>{{t "chat.note.on_demand"}}</p>
<div id="log" class="chat-log" aria-live="polite"></div>
<form id="composer" class="chat-composer">
<textarea id="input" rows="3" placeholder="Ask something…" required></textarea>
<button type="submit" id="send">Send</button>
<textarea id="input" rows="3" placeholder="{{t "chat.input_placeholder"}}" required></textarea>
<button type="submit" id="send">{{t "chat.send"}}</button>
</form>
<p class="note">The conversation is kept in this browser, so a reload does not lose it;
"New conversation" clears it. Nothing of it is stored on the gateway — what is
recorded there is the same usage any request records: tokens, cost and latency.</p>
<p class="note">{{t "chat.kept"}}</p>
{{else}}
<p class="empty">No models are available to you right now.</p>
<p class="note">Either nothing is placed yet, or your project's allowlist does
not include anything that is. <code>/admin</code> shows the cluster if you have
access to it.</p>
<p class="empty">{{t "chat.empty"}}</p>
<p class="note">{{th "chat.empty_why"}}</p>
{{end}}
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div>
<script>
// What the script says, in the page's language. The template writes each
// text as a JavaScript string; {name} marks where a value goes, so a
// language can put it where its grammar wants it.
const say = {
preparing: {{t "chat.js.preparing"}},
downloadFailed: {{t "chat.js.download_failed"}},
stageWaiting: {{t "chat.js.stage.waiting"}},
stageDownloading: {{t "chat.js.stage.downloading"}},
stageStarting: {{t "chat.js.stage.starting"}},
stageReady: {{t "chat.js.stage.ready"}},
of: {{t "chat.js.of"}},
couldNotStart: {{t "chat.js.could_not_start"}},
tokens: {{t "chat.js.tokens"}},
tokPerS: {{t "chat.js.tok_per_s"}},
firstToken: {{t "chat.js.first_token"}},
seconds: {{t "chat.js.seconds"}},
stopped: {{t "chat.js.stopped"}},
stop: {{t "chat.js.stop"}},
send: {{t "chat.send"}},
refused: {{t "chat.js.refused"}},
downloadZip: {{t "chat.js.download_zip"}},
stoppedByPolicy: {{t "chat.js.stopped_by_policy"}},
cutOff: {{t "chat.js.cut_off"}},
stoppedByPolicyEmpty: {{t "chat.js.stopped_by_policy_empty"}},
stoppedEmpty: {{t "chat.js.stopped_empty"}},
noContent: {{t "chat.js.no_content"}},
decimal: {{t "_meta.decimal"}},
};
// fill puts values into a text from say.
function fill(s, vals) { return s.replace(/\{(\w+)\}/g, (m, k) => (k in vals ? String(vals[k]) : m)); }
// fixed is toFixed with the language's decimal separator.
function fixed(n, places) { return n.toFixed(places).replace(".", say.decimal); }
// The policy a guardrail acted by, when the response says.
function policySuffix(resp) {
const p = resp.headers.get("x-llmgw-guard-policy");
@@ -111,7 +134,7 @@ access to it.</p>
system: sys.value, temperature: temp.value }));
} catch (_) { /* no storage: nothing kept */ }
};
temp.addEventListener("input", () => { tempOut.textContent = (+temp.value).toFixed(1); save(); });
temp.addEventListener("input", () => { tempOut.textContent = fixed(+temp.value, 1); save(); });
sys.addEventListener("change", save);
document.getElementById("reset").onclick = () => { messages = []; log.textContent = ""; save(); };
@@ -214,7 +237,7 @@ access to it.</p>
async function downloadCode(reply, btn) {
const was = btn.textContent;
btn.disabled = true;
btn.textContent = "Preparing…";
btn.textContent = say.preparing;
try {
const resp = await fetch("/chat/download", {
method: "POST",
@@ -234,7 +257,7 @@ access to it.</p>
URL.revokeObjectURL(url);
btn.textContent = was;
} catch (err) {
btn.textContent = "download failed";
btn.textContent = say.downloadFailed;
} finally {
btn.disabled = false;
}
@@ -242,12 +265,12 @@ access to it.</p>
const ctxValue = () => (document.getElementById("context") || {}).value || "";
const stages = {
waiting: "Waiting for a node with room",
downloading: "Downloading weights",
starting: "Starting the engine",
ready: "Ready",
waiting: say.stageWaiting,
downloading: say.stageDownloading,
starting: say.stageStarting,
ready: say.stageReady,
};
function mib(n) { return (n / 1048576).toFixed(0) + " MiB"; }
function mib(n) { return fixed(n / 1048576, 0) + " MiB"; }
// Loads a model that is not running and shows how far along it is, in the
// reply's own bubble. Resolves true once it answers, false if it cannot.
@@ -277,8 +300,8 @@ access to it.</p>
st = await resp.json();
for (;;) {
const secs = Math.round((Date.now() - started) / 1000);
let text = (stages[st.state] || st.state) + " — " + secs + "s";
if (st.state === "downloading" && st.total) text += " · " + mib(st.done) + " of " + mib(st.total);
let text = (stages[st.state] || st.state) + " — " + fill(say.seconds, { n: secs });
if (st.state === "downloading" && st.total) text += " · " + fill(say.of, { done: mib(st.done), total: mib(st.total) });
label.textContent = text;
// Indeterminate (no value) unless there is a real number: a bar that
// sits at 0% for the engine start says it is stuck when it is not.
@@ -288,7 +311,7 @@ access to it.</p>
// control plane places its replacement, so one sighting is not final.
failures = st.state === "failed" ? failures + 1 : 0;
if (st.state === "stopped" || failures >= 5) {
throw new Error(st.message || "the model could not be started");
throw new Error(st.message || say.couldNotStart);
}
await new Promise((r) => setTimeout(r, 1500));
const r = await fetch("/chat/model-status?model=" + encodeURIComponent(name) +
@@ -314,11 +337,11 @@ access to it.</p>
// What answered: a model with replicas on a card and on CPUs is served
// by either, and the speed below means nothing without knowing which.
if (hw) parts.push(hw === "cpu" ? "CPU" : "GPU " + hw);
if (tokens) parts.push(tokens + " tokens");
if (tokens && tFirst && tEnd > tFirst) parts.push((tokens / ((tEnd - tFirst) / 1000)).toFixed(1) + " tok/s");
if (tFirst) parts.push("first token " + ((tFirst - t0) / 1000).toFixed(2) + " s");
parts.push(((tEnd - t0) / 1000).toFixed(1) + " s");
if (stopped) parts.push("stopped");
if (tokens) parts.push(fill(say.tokens, { n: tokens }));
if (tokens && tFirst && tEnd > tFirst) parts.push(fill(say.tokPerS, { n: fixed(tokens / ((tEnd - tFirst) / 1000), 1) }));
if (tFirst) parts.push(fill(say.firstToken, { n: fixed((tFirst - t0) / 1000, 2) }));
parts.push(fill(say.seconds, { n: fixed((tEnd - t0) / 1000, 1) }));
if (stopped) parts.push(say.stopped);
p.textContent = parts.join(" · ");
out.after(p);
}
@@ -340,7 +363,7 @@ access to it.</p>
let tokens = 0, pieces = 0, stopped = false;
const opt = model.selectedOptions[0];
busy = new AbortController();
send.textContent = "Stop";
send.textContent = say.stop;
send.classList.add("stop");
const t0 = performance.now();
let tFirst = 0;
@@ -377,7 +400,7 @@ access to it.</p>
const e = JSON.parse(text).error;
if (e && e.code === "content_policy_violation") {
const policy = resp.headers.get("x-llmgw-guard-policy");
shown = "Refused by the gateway's policy" + (policy ? " " + policy : "") + ": " + e.message;
shown = fill(say.refused, { policy: policy ? " " + policy : "", message: e.message });
} else if (e && e.code === "no_store") {
shown = e.message;
}
@@ -437,14 +460,14 @@ access to it.</p>
const dl = document.createElement("button");
dl.type = "button";
dl.className = "link download";
dl.textContent = "Download as .zip";
dl.textContent = say.downloadZip;
dl.onclick = () => downloadCode(reply, dl);
out.after(dl);
}
if (finish === "content_filter") {
const note = document.createElement("p");
note.className = "note truncated";
note.textContent = "Stopped by the gateway's policy" + policySuffix(resp) + ".";
note.textContent = fill(say.stoppedByPolicy, { policy: policySuffix(resp) });
out.after(note);
}
if (finish === "length") {
@@ -452,7 +475,7 @@ access to it.</p>
// placed with textContent, and this is ours.
const note = document.createElement("p");
note.className = "note truncated";
note.textContent = "Cut off at the length limit — ask it to continue.";
note.textContent = say.cutOff;
out.after(note);
}
stats(out, tokens || pieces, t0, tFirst, performance.now(), stopped, resp.headers.get("x-llmgw-hardware"));
@@ -462,14 +485,14 @@ access to it.</p>
renderMath(out);
}
else if (finish === "content_filter") {
out.textContent = "(stopped by the gateway's policy" + policySuffix(resp) + " before anything was shown)";
out.textContent = fill(say.stoppedByPolicyEmpty, { policy: policySuffix(resp) });
out.classList.add("failed"); messages.pop();
}
else if (stopped) { out.textContent = "(stopped before anything came back)"; out.classList.add("failed"); messages.pop(); }
else { out.textContent = "(no content came back)"; out.classList.add("failed"); }
else if (stopped) { out.textContent = say.stoppedEmpty; out.classList.add("failed"); messages.pop(); }
else { out.textContent = say.noContent; out.classList.add("failed"); }
} catch (err) {
if (err.name === "AbortError") {
out.textContent = "(stopped before anything came back)";
out.textContent = say.stoppedEmpty;
messages.pop();
} else {
out.textContent = "(" + err + ")";
@@ -477,7 +500,7 @@ access to it.</p>
out.classList.add("failed");
} finally {
busy = null;
send.textContent = "Send";
send.textContent = say.send;
send.classList.remove("stop");
save();
input.focus();
@@ -490,7 +513,7 @@ access to it.</p>
const kept = load_();
if (!kept) return;
if (kept.system) sys.value = kept.system;
if (kept.temperature != null) { temp.value = kept.temperature; tempOut.textContent = (+temp.value).toFixed(1); }
if (kept.temperature != null) { temp.value = kept.temperature; tempOut.textContent = fixed(+temp.value, 1); }
// Open when there is a prompt in it, so it is not a hidden surprise.
document.getElementById("settings").open = !!kept.system;
if (kept.model && [...model.options].some((o) => o.value === kept.model)) { model.value = kept.model; showSimNote(); }
+14 -18
View File
@@ -1,20 +1,20 @@
{{define "claim"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="robots" content="noindex, nofollow">
<title>Your llmgw key</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "claim.title"}}</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>Your API key</h1>
<p class="sub">llmgw · {{.BaseURL}}</p>
<h1>{{t "claim.heading"}}</h1>
<p class="sub" translate="no">llmgw · {{.BaseURL}}</p>
</header>
{{if .Secret}}
<p>This is shown once. Nothing can produce it again — only its hash was kept.</p>
<p>{{t "claim.shown_once"}}</p>
<pre>{{.Secret}}</pre>
<p>Store it in a password manager now, then close this page.</p>
<p>{{t "claim.store_it"}}</p>
<h2>Using it</h2>
<h2>{{t "claim.using_it"}}</h2>
<pre>export LLMGW_API_KEY={{.Secret}}
curl {{.BaseURL}}/v1/chat/completions \
@@ -22,27 +22,23 @@ curl {{.BaseURL}}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "&lt;model-name&gt;", "messages": [{"role":"user","content":"hello"}]}'</pre>
<p class="note">
Replace &lt;model-name&gt; with a name from
<code>GET {{.BaseURL}}/v1/models</code>, which needs your key. Sending the
placeholder unchanged is answered with the list of models that are available.
{{th "claim.replace_placeholder" .BaseURL}}
</p>
<pre>from openai import OpenAI
client = OpenAI(base_url="{{.BaseURL}}/v1", api_key=os.environ["LLMGW_API_KEY"])</pre>
<p class="note"><code>GET {{.BaseURL}}/v1/models</code> lists what this key may use.
<code>GET {{.BaseURL}}/me</code> shows its limits and what it has spent.</p>
<p class="note">{{th "claim.endpoints" .BaseURL}}</p>
{{else if .Gone}}
<p>{{.Message}}</p>
<p class="note">A link works once and then expires. Ask whoever sent it for a new one.</p>
<p class="note">{{t "claim.ask_again"}}</p>
{{else}}
<p>A key was issued for you. Revealing it uses up this link, and the key is
shown only this once.</p>
<p>{{t "claim.issued"}}</p>
<form method="post">
<button type="submit">Reveal my key</button>
<button type="submit">{{t "claim.reveal"}}</button>
</form>
<p class="note">Expires {{.ExpiresAt}}.</p>
<p class="note">{{t "claim.expires" .ExpiresAt}}</p>
{{end}}
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}
+120 -125
View File
@@ -1,19 +1,19 @@
{{define "dashboard"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<noscript><meta http-equiv="refresh" content="{{.RefreshSec}}"></noscript>
<noscript>{{.RefreshMeta}}</noscript>
<meta name="htmx-config" content='{"allowEval":false}'>
<script src="/_static/htmx-2.0.11.min.js"></script>
<script defer src="/_static/pages.js"></script>
<title>llmgw — cluster</title><link rel="stylesheet" href="{{stylesheet}}">
<title>llmgw — {{t "dash.title"}}</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>Cluster</h1>
<p class="sub">{{.NodeCount}} nodes · {{.ReadyReplicas}} of {{.TotalReplicas}} replicas ready · {{.ModelCount}} models</p>
<h1>{{t "dash.h1"}}</h1>
<p class="sub">{{t "dash.sub" .NodeCount .ReadyReplicas .TotalReplicas .ModelCount}}</p>
{{if .OnAdminListener}}<nav>
<a href="/metrics">metrics</a>
<a href="/debug/pprof/">pprof</a>
<a href="/api/v1/nodes">api</a>
<a href="/metrics">{{t "dash.nav.metrics"}}</a>
<a href="/debug/pprof/" translate="no">pprof</a>
<a href="/api/v1/nodes" translate="no">api</a>
</nav>{{else}}{{template "nav" .Menu}}{{end}}
</header>
@@ -24,16 +24,16 @@
<div id="live" hx-get="{{if .OnAdminListener}}/{{else}}/admin{{end}}" hx-trigger="every {{.RefreshSec}}s" data-visible-only hx-select="#live" hx-swap="outerHTML">
{{if .HasCharts}}
<div class="tiles">
<div class="tile"><div class="v">{{.Requests24h}}</div><div class="k">requests · 24h</div>{{.RequestSpark}}</div>
<div class="tile"><div class="v">{{compact .Tokens24h}}</div><div class="k">tokens · 24h</div>{{.TokenSpark}}</div>
<div class="tile"><div class="v">{{.Errors24h}}</div><div class="k">errors · 24h</div></div>
<div class="tile"><div class="v">{{units .Spend24h}}</div><div class="k">spend · 24h</div></div>
<div class="tile"><div class="v">{{count .Requests24h}}</div><div class="k">{{t "dash.tile.requests"}}</div>{{.RequestSpark}}</div>
<div class="tile"><div class="v">{{compact .Tokens24h}}</div><div class="k">{{t "dash.tile.tokens"}}</div>{{.TokenSpark}}</div>
<div class="tile"><div class="v">{{count .Errors24h}}</div><div class="k">{{t "dash.tile.errors"}}</div></div>
<div class="tile"><div class="v">{{units .Spend24h}}</div><div class="k">{{t "dash.tile.spend"}}</div></div>
</div>
{{end}}
<h2>Capacity</h2>
<h2>{{t "dash.capacity"}}</h2>
{{if .Summary.GPUs}}
<table><tr><th>Class</th><th class="num">In use</th><th class="num">Free</th><th class="num">Faulty</th><th>Usage</th></tr>
<table><tr><th>{{t "dash.cap.class"}}</th><th class="num">{{t "dash.cap.in_use"}}</th><th class="num">{{t "dash.cap.free"}}</th><th class="num">{{t "dash.cap.faulty"}}</th><th>{{t "dash.cap.usage"}}</th></tr>
{{range .Summary.GPUs}}<tr>
<td><code>{{.Class}}</code></td>
<td class="num">{{.Used}}/{{.Total}}</td>
@@ -45,74 +45,78 @@
{{/* No GPUs is not no capacity: a CPU fleet still has memory and disk,
which is what decides what it can hold. */}}
<div class="capacity-cpu">
<div><b>{{.CPU.Ready}}/{{.CPU.Nodes}}</b> <span>nodes ready · CPU only</span></div>
<div><b>{{.CPU.MemoryGB}} GB</b> <span>memory</span></div>
{{if .CPU.DiskTotal}}<div><b>{{.CPU.DiskFreeGB}} of {{.CPU.DiskTotal}} GB</b> <span>free for weights</span></div>{{end}}
<div><b>{{.CPU.Ready}}/{{.CPU.Nodes}}</b> <span>{{t "dash.cpu.ready"}}</span></div>
<div><b>{{count .CPU.MemoryGB}} GB</b> <span>{{t "dash.cpu.memory"}}</span></div>
{{if .CPU.DiskTotal}}<div><b>{{t "dash.cpu.disk_of" (count .CPU.DiskFreeGB) (count .CPU.DiskTotal)}}</b> <span>{{t "dash.cpu.disk_free"}}</span></div>{{end}}
</div>
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
{{else}}<p class="empty">{{t "dash.no_nodes"}}</p>{{end}}
{{with .GPUMap}}
<h2>GPUs <span class="opt">by
<h2>{{t "dash.gpumap.title"}} <span class="opt">{{t "dash.gpumap.by"}}
{{range .Modes}}{{if eq .Key $.GPUMap.By}}<strong>{{.Label}}</strong>{{else}}<a href="?gpumap={{.Key}}">{{.Label}}</a>{{end}} {{end}}</span></h2>
<div class="gpu-map">
{{range .Rows}}<div class="gpu-row"><span class="gpu-node">{{.Node}}</span>{{range .Cells}}<span class="gpu-cell" style="{{.Style}}" title="{{.Title}}">{{.Label}}</span>{{end}}</div>
{{end}}</div>
<p class="note">{{.Legend}} A red outline is a card wearing out — XID errors, memory rows remapped or waiting for a reset, NVLink errors. The figure is the one coloured by; the rest of a card is under the pointer.</p>
<p class="note">{{.Legend}} {{t "dash.gpumap.note"}}</p>
{{end}}
{{with .Efficiency}}
<h3>What each model gets from its cards <span class="opt">last 15 minutes</span></h3>
<h3>{{t "dash.eff.title"}} <span class="opt">{{t "dash.eff.window"}}</span></h3>
{{/* Per card, because a model on eight cards making twice the tokens of one
on two is the worse of the two; per watt-hour, because that is what the
room is paying. */}}
<table class="rows"><tr><th>Model</th><th class="num">GPUs</th><th class="num">Output tokens/s</th><th class="num">Per GPU</th><th class="num">Busy</th><th class="num">Draw</th><th class="num">Tokens per Wh</th></tr>
{{range .}}<tr><td>{{.Model}}</td><td class="num">{{printf "%.0f" .GPUs}}</td><td class="num">{{printf "%.1f" .TokensPerSec}}</td>
<td class="num">{{printf "%.1f" .PerGPU}}</td><td class="num">{{printf "%.0f" .Util}}%</td>
<td class="num">{{if .Watts}}{{printf "%.0f" .Watts}} W{{else}}—{{end}}</td><td class="num">{{if .PerWh}}{{printf "%.0f" .PerWh}}{{else}}—{{end}}</td></tr>
<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th class="num">GPUs</th><th class="num">{{t "dash.eff.tps"}}</th><th class="num">{{t "dash.eff.per_gpu"}}</th><th class="num">{{t "dash.eff.busy"}}</th><th class="num">{{t "dash.eff.draw"}}</th><th class="num">{{t "dash.eff.per_wh"}}</th></tr>
{{range .}}<tr><td>{{.Model}}</td><td class="num">{{num .GPUs 0}}</td><td class="num">{{num .TokensPerSec 1}}</td>
<td class="num">{{num .PerGPU 1}}</td><td class="num">{{num .Util 0}}%</td>
<td class="num">{{if .Watts}}{{num .Watts 0}} W{{else}}—{{end}}</td><td class="num">{{if .PerWh}}{{num .PerWh 0}}{{else}}—{{end}}</td></tr>
{{end}}</table>
{{end}}
{{with .IdleGPUs}}
<h3>Cards held and doing nothing</h3>
<h3>{{t "dash.idle.title"}}</h3>
{{/* The first place to find room: replicas whose every card reads under 5%
busy now. A model kept warm on purpose is here too; whether it should
be is the operator's call. */}}
<table class="rows"><tr><th>Model</th><th>Replica</th><th>Node</th><th class="num">GPUs</th><th class="num">Busiest card</th><th>Last request</th></tr>
<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th>{{t "dash.col.replica"}}</th><th>{{t "dash.col.node"}}</th><th class="num">GPUs</th><th class="num">{{t "dash.idle.busiest"}}</th><th>{{t "dash.idle.last_request"}}</th></tr>
{{range .}}<tr><td>{{.Model}}</td><td class="clip"><code>{{.Replica}}</code></td><td><code>{{.Node}}</code></td>
<td class="num">{{len .GPUs}}</td><td class="num">{{printf "%.0f" .MaxUtil}}%</td><td>{{if .Idle}}{{.Idle}} ago{{else}}<span class="note">not known</span>{{end}}</td></tr>
<td class="num">{{len .GPUs}}</td><td class="num">{{num .MaxUtil 0}}%</td><td>{{if .Idle}}{{t "dash.idle.ago" .Idle}}{{else}}<span class="note">{{t "dash.idle.unknown"}}</span>{{end}}</td></tr>
{{end}}</table>
{{end}}
<h2>Nodes</h2>
<h2>{{t "dash.nodes"}}</h2>
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
{{if .Nodes}}
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyGPUs}}<th>vLLM build</th>{{end}}{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
<table class="nodes rows"><tr><th>{{t "dash.col.node"}}</th><th>{{t "dash.col.state"}}</th><th>CPU</th><th>{{t "dash.nodes.ram"}}</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">{{t "dash.nodes.disk"}}</th><th>{{t "dash.nodes.engines"}}</th>{{if .AnyGPUs}}<th>{{t "dash.nodes.build"}}</th>{{end}}{{if .AnyClasses}}<th>{{t "dash.nodes.classes"}}</th>{{end}}{{if .AnyFabrics}}<th>{{t "dash.nodes.fabrics"}}</th>{{end}}<th class="num">{{t "dash.nodes.models"}}</th><th class="num">{{t "dash.nodes.replicas"}}</th><th>{{t "dash.nodes.seen"}}</th></tr>
{{range .Nodes}}<tr>
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
<td class="state state-{{.State}}">{{.State}}</td>
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">{{t "dash.nodes.logs"}}</a></td>
{{/* The state is a class as well as a word: the class stays, the word
is the reader's. */}}
<td class="state state-{{.State}}">{{t (print "dash.state." .State)}}</td>
{{/* A dash, not an empty bar, when the agent has not measured it lately:
unknown is not idle. */}}
<td class="load">{{with .Load}}{{template "minibar" pctbar .CPUPercent}}<span class="load-v">{{printf "%.0f" .CPUPercent}}%{{if .CPUs}} <span class="note">of {{.CPUs}}</span>{{end}}</span>{{else}}—{{end}}</td>
<td class="load">{{with .Load}}{{template "minibar" pctbar .MemPercent}}<span class="load-v">{{bytes .MemUsed}} <span class="note">of {{bytes .MemTotal}}</span></span>{{else}}{{if .MemoryGB}}{{.MemoryGB}}G{{else}}—{{end}}{{end}}</td>
<td class="load">{{with .Load}}{{template "minibar" pctbar .CPUPercent}}<span class="load-v">{{num .CPUPercent 0}}%{{if .CPUs}} <span class="note">{{t "dash.nodes.cpus_of" .CPUs}}</span>{{end}}</span>{{else}}—{{end}}</td>
<td class="load">{{with .Load}}{{template "minibar" pctbar .MemPercent}}<span class="load-v">{{bytes .MemUsed}} <span class="note">{{t "dash.nodes.mem_of" (bytes .MemTotal)}}</span></span>{{else}}{{if .MemoryGB}}{{.MemoryGB}}G{{else}}—{{end}}{{end}}</td>
{{if $.AnyGPUs}}<td class="num">{{if .GPUTotal}}{{.GPUFree}}/{{.GPUTotal}}{{else}}—{{end}}</td>{{end}}
{{/* Free out of total, because the first is meaningless without the
second, and an em dash where nothing was reported: an agent that
was never asked about its disk must not read as a full one. */}}
<td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">{{t "dash.nodes.no_engines"}}</span>{{end}}</td>
{{/* Which vLLM build the node installs: left to its hardware, or chosen.
What is installed beside it, so a choice and its outcome are read
together. Only a GPU node has one; changing it is for an
administrator, on the page reached through a session. */}}
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{t (print "dash.state." .State)}}</span>{{end}}{{end}}
{{if and $.Menu.CSRF .EngineBuilds}}
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
<input type="hidden" name="engine" value="vllm">
<select name="build" title="the vLLM build this node installs">
<select name="build" title="{{t "dash.nodes.build_hint"}}">
{{range engineBuilds "vllm"}}<option value="{{.}}"{{if eq . (index $node.EngineBuilds "vllm")}} selected{{end}}>{{buildLabel .}}</option>{{end}}
</select>
<noscript><button type="submit">Save</button></noscript>
{{/* For a browser without script; the script below hides it. Not in a
<noscript>, whose inside is text to the catalogue check. */}}
<button type="submit">{{t "common.save"}}</button>
</form>
{{end}}
{{else}}—{{end}}</td>{{end}}
@@ -121,42 +125,47 @@
<td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
<td class="num">{{.Replicas}}</td>
<td>{{or .LastSeenAgo "—"}}</td>
{{/* What is unhealthy, what an install said and the node's warnings come
from the control plane and the agent, in English. */}}
</tr>{{if .Unhealthy}}<tr><td colspan="12" class="note bad">{{join .Unhealthy}}</td></tr>{{end}}
{{range .Provision}}{{if and (ne .State "ready") (ne .State "skipped")}}<tr><td colspan="12" class="note{{if or (eq .State "failed") (eq .State "unsupported")}} bad{{end}}">{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{.State}}: {{.Detail}}</td></tr>{{end}}{{end}}
{{range .Provision}}{{if and (ne .State "ready") (ne .State "skipped")}}<tr><td colspan="12" class="note{{if or (eq .State "failed") (eq .State "unsupported")}} bad{{end}}">{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{t (print "dash.state." .State)}}: {{.Detail}}</td></tr>{{end}}{{end}}
{{range .Warnings}}<tr><td colspan="12" class="note bad">{{.}}</td></tr>{{end}}
{{end}}</table>
<script>
// A build chosen is saved as it is chosen, with the page reloaded on the
// answer: one select, no separate button to forget.
document.querySelectorAll("form.build-choice select").forEach(function (s) {
s.form.querySelectorAll("button").forEach(function (b) { b.hidden = true; });
s.addEventListener("change", function () { s.form.submit(); });
});
</script>
{{else}}<p class="empty">No agents have registered.</p>{{end}}
{{else}}<p class="empty">{{t "dash.no_agents"}}</p>{{end}}
<h2>Models</h2>
<h2>{{t "dash.models"}}</h2>
{{if .Models}}
<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Class</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th>{{t "dash.models.aliases"}}</th><th class="num">{{t "dash.models.ready"}}</th><th class="num">{{t "dash.models.wanted"}}</th><th>{{t "dash.models.class"}}</th><th>{{t "dash.models.reserves"}}</th><th>{{t "dash.models.profiles"}}</th>{{if .AnyFallbacks}}<th>{{t "dash.models.fallbacks"}}</th>{{end}}<th>{{t "dash.models.for"}}</th></tr>
{{range .Models}}<tr>
<td><code>{{.Name}}</code></td>
<td>{{join .Aliases}}</td>
<td class="num">{{.Ready}}</td>
<td class="num">{{.Min}}–{{.Max}}</td>
<td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
<td{{if .HeldFor}} class="warn" title="{{t "dash.models.held_for" .HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="{{t "dash.models.preempted"}}">−{{.Preempted}}</span>{{end}}</td>
{{/* The warnings are the catalogue's own, in English. */}}
<td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
{{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
<td>{{if .External}}<span class="tag" title="served outside the cluster; its prompts leave it">external</span> {{.External}}{{else}}{{join .Profiles}}{{end}}</td>
{{- if .MeasuredMemoryGB}} <span class="note">{{t "dash.models.used" (num .MeasuredMemoryGB 1)}}</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
<td>{{if .External}}<span class="tag" title="{{t "dash.models.external_hint"}}">{{t "dash.models.external"}}</span> {{.External}}{{else}}{{join .Profiles}}{{end}}</td>
{{if $.AnyFallbacks}}<td>{{join .Fallbacks}}</td>{{end}}
{{/* Cut to the column, the whole sentence on hover: a row of its own for
each description made the table twice as long as the catalog. */}}
<td class="clip"{{with .Description}} title="{{.}}"{{end}}>{{or .Description "—"}}</td>
</tr>
{{end}}</table>
{{else}}<p class="empty">The catalog is empty.</p>{{end}}
{{else}}<p class="empty">{{t "dash.models.empty"}}</p>{{end}}
{{if .Unplaced}}
<h2>Why nothing was placed</h2>
<h2>{{t "dash.unplaced"}}</h2>
{{/* The scheduler's reasons, as it gives them. */}}
{{range .Unplaced}}
<p><code>{{.Model}}</code></p>
<pre>{{range .Rejections}}{{.Profile}} @ {{or .Pool "-"}}{{if .Node}}/{{.Node}}{{end}}: {{.Reason}}
@@ -164,181 +173,167 @@ document.querySelectorAll("form.build-choice select").forEach(function (s) {
{{end}}
{{end}}
<h2>Replicas</h2>
<h2>{{t "dash.replicas"}}</h2>
{{if .Replicas}}
<table class="rows"><tr><th>Replica</th><th>Model</th><th>State</th><th>Pool</th><th>Endpoint</th><th class="num">Age</th><th>Nodes</th></tr>
<table class="rows"><tr><th>{{t "dash.col.replica"}}</th><th>{{t "dash.col.model"}}</th><th>{{t "dash.col.state"}}</th><th>{{t "dash.replicas.pool"}}</th><th>{{t "dash.replicas.endpoint"}}</th><th class="num">{{t "dash.replicas.age"}}</th><th>{{t "dash.nodes"}}</th></tr>
{{range .Replicas}}<tr>
<td><code>{{.ID}}</code></td>
<td>{{.Model}}</td>
<td class="state state-{{.State}}">{{.State}}</td>
<td class="state state-{{.State}}">{{t (print "dash.state." .State)}}</td>
<td>{{or .Pool "—"}}</td>
<td>{{if .Endpoint}}<code>{{.Endpoint}}</code>{{else}}—{{end}}</td>
<td class="num">{{duration .AgeSec}}</td>
<td class="num">{{age .AgeSec}}</td>
<td>{{range .Shards}}<code title="{{.Node}}">{{shortNode .Node $.NodePrefix}}</code> {{end}}</td>
</tr>{{if .Err}}<tr><td colspan="7" class="note">{{.Err}}</td></tr>{{end}}
{{end}}</table>
{{else}}<p class="empty">No replicas.</p>{{end}}
{{else}}<p class="empty">{{t "dash.replicas.none"}}</p>{{end}}
{{if .Summary.IdleReplicas}}
<h2>Holding hardware nothing is asking for</h2>
<table><tr><th>Replica</th><th>Model</th><th class="num">GPUs</th><th class="num">Unused for</th></tr>
<h2>{{t "dash.holding"}}</h2>
<table><tr><th>{{t "dash.col.replica"}}</th><th>{{t "dash.col.model"}}</th><th class="num">GPUs</th><th class="num">{{t "dash.holding.unused_for"}}</th></tr>
{{range .Summary.IdleReplicas}}<tr>
<td><code>{{.ID}}</code></td>
<td>{{.Model}}</td>
<td class="num">{{.GPUs}}</td>
<td class="num">{{duration .IdleSec}}</td>
<td class="num">{{age .IdleSec}}</td>
</tr>{{end}}</table>
{{end}}
{{if or .Summary.Fabrics (and .Fabrics (or .Fabrics.Physical .Fabrics.Problems))}}
<h2>Fabrics</h2>
<h2>{{t "dash.nodes.fabrics"}}</h2>
{{with .Fabrics}}{{if .Problems}}
<ul class="fabric-problems">{{range .Problems}}
<li class="{{.Level}}"><span class="tag{{if eq (printf "%s" .Level) "fail"}} bad{{end}}">{{.Level}}</span> {{.Message}}</li>{{end}}
<li class="{{.Level}}"><span class="tag{{if eq (printf "%s" .Level) "fail"}} bad{{end}}">{{t (print "fabric.level." .Level)}}</span> {{.Message}}</li>{{end}}
</ul>{{end}}{{end}}
{{if .Summary.Fabrics}}
<table><tr><th>Fabric</th><th>Is physically</th><th class="num">Nodes</th><th class="num">Fully free</th><th>Weights</th></tr>
<table><tr><th>{{t "dash.fabrics.fabric"}}</th><th>{{t "dash.fabrics.physically"}}</th><th class="num">{{t "dash.nodes"}}</th><th class="num">{{t "dash.fabrics.fully_free"}}</th><th>{{t "dash.fabrics.weights"}}</th></tr>
{{range .Summary.Fabrics}}{{$name := .Fabric}}<tr>
<td><code>{{.Fabric}}</code></td>
<td>{{with $.Fabrics}}{{range .Physical}}{{if has .Declared $name}}<div>{{.Label}}{{if .RateGbps}} <span class="note">· {{.RateGbps}} Gb/s</span>{{end}}</div>{{end}}{{end}}{{end}}</td>
<td>{{with $.Fabrics}}{{range .Physical}}{{if has .Declared $name}}<div>{{.Label}}{{if .RateGbps}} <span class="note">· {{t "fabric.gbps" .RateGbps}}</span>{{end}}</div>{{end}}{{end}}{{end}}</td>
<td class="num">{{.Nodes}}</td>
<td class="num">{{.LargestFreeBlock}}</td>
<td>{{if .NodesWithWeights}}{{.Models}} model(s) on {{.NodesWithWeights}}/{{.Nodes}} nodes{{else}}<span class="note">not reported</span>{{end}}</td>
<td>{{if .NodesWithWeights}}{{t "dash.fabrics.weights_on" .Models .NodesWithWeights .Nodes}}{{else}}<span class="note">{{t "dash.fabrics.not_reported"}}</span>{{end}}</td>
</tr>{{end}}</table>
{{end}}
{{with .Fabrics}}{{if .Physical}}
<p><a href="{{if $.OnAdminListener}}/fabrics{{else}}/admin/fabrics{{end}}">See each fabric's map →</a></p>
<h3>Subnet managers</h3>
<table><tr><th>Fabric</th><th>Is physically</th><th>Master</th><th>Standby</th><th class="num">Nodes</th><th>Last change of master</th></tr>
<p><a href="{{if $.OnAdminListener}}/fabrics{{else}}/admin/fabrics{{end}}">{{t "dash.fabrics.see_map"}}</a></p>
<h3>{{t "dash.fabrics.sms"}}</h3>
<table><tr><th>{{t "dash.fabrics.fabric"}}</th><th>{{t "dash.fabrics.physically"}}</th><th>{{t "dash.fabrics.master"}}</th><th>{{t "dash.fabrics.standby"}}</th><th class="num">{{t "dash.nodes"}}</th><th>{{t "dash.fabrics.last_change"}}</th></tr>
{{range .Physical}}<tr>
<td>{{if .Declared}}{{range $i, $d := .Declared}}{{if $i}}, {{end}}<code>{{$d}}</code>{{end}}{{else}}<span class="note">undeclared</span>{{end}}</td>
<td>{{.Label}}{{if .RateGbps}} <span class="note">· {{.RateGbps}} Gb/s</span>{{end}}</td>
<td>{{if eq .Kind "roce"}}<span class="note">none: RoCE</span>{{else}}{{with .SM}}<code>{{.GUID}}</code>{{if .LID}} <span class="note">LID {{.LID}}, priority {{.Priority}}</span>{{end}}{{else}}—{{end}}{{end}}</td>
<td>{{if eq .Kind "roce"}}—{{else if .SMs}}{{if .Standbys}}{{.Standbys}}{{else}}<span class="bad">none</span>{{end}}{{else}}<span class="note">not listed</span>{{end}}</td>
<td>{{if .Declared}}{{range $i, $d := .Declared}}{{if $i}}, {{end}}<code>{{$d}}</code>{{end}}{{else}}<span class="note">{{t "fabric.undeclared"}}</span>{{end}}</td>
<td>{{.Label}}{{if .RateGbps}} <span class="note">· {{t "fabric.gbps" .RateGbps}}</span>{{end}}</td>
<td>{{if eq .Kind "roce"}}<span class="note">{{t "dash.fabrics.roce"}}</span>{{else}}{{with .SM}}<code>{{.GUID}}</code>{{if .LID}} <span class="note">{{t "dash.fabrics.lid_priority" .LID .Priority}}</span>{{end}}{{else}}—{{end}}{{end}}</td>
<td>{{if eq .Kind "roce"}}—{{else if .SMs}}{{if .Standbys}}{{.Standbys}}{{else}}<span class="bad">{{t "dash.fabrics.none"}}</span>{{end}}{{else}}<span class="note">{{t "dash.fabrics.not_listed"}}</span>{{end}}</td>
<td class="num">{{len .Members}}</td>
<td>{{with .Changes}}{{with index . 0}}{{ago .At}}: <code>{{.From}}</code> → <code>{{.To}}</code>{{end}}{{if gt (len .) 1}} <span class="note">({{len .}} in a day)</span>{{end}}{{else}}<span class="note">none in a day</span>{{end}}</td>
<td>{{with .Changes}}{{with index . 0}}{{ago .At}}: <code>{{.From}}</code> → <code>{{.To}}</code>{{end}}{{if gt (len .) 1}} <span class="note">{{t "dash.fabrics.in_a_day" (len .)}}</span>{{end}}{{else}}<span class="note">{{t "dash.fabrics.none_in_a_day"}}</span>{{end}}</td>
</tr>{{end}}</table>
{{end}}{{end}}
<p class="note">A fabric's name is what the nodes were told; what it is physically is
the subnet manager its InfiniBand ports see, or the network its RoCE ports are on.
A replica spanning nodes is only ever formed on one of those. Fully free counts whole
unused nodes, which is what decides whether such a replica can land.</p>
<p class="note">{{t "dash.fabrics.note"}}</p>
{{end}}
{{with .NVLink}}
<h2>NVLink domains</h2>
<p>{{len .Domains}} multi-node domain{{if ne (len .Domains) 1}}s{{end}}{{range .Domains}} · <code>{{shortDomain .Key}}</code> {{.Usable}}/{{len .Members}} trays usable{{end}}
{{if .Problems}} · <span class="bad">{{len .Problems}} problem{{if ne (len .Problems) 1}}s{{end}}</span>{{end}}
· <a href="{{if $.OnAdminListener}}/nvlink{{else}}/admin/nvlink{{end}}">see the racks →</a></p>
<h2>{{t "dash.nvlink.title"}}</h2>
<p>{{tn "dash.nvlink.domains" (len .Domains)}}{{range .Domains}} · <code>{{shortDomain .Key}}</code> {{t "dash.nvlink.trays" .Usable (len .Members)}}{{end}}
{{if .Problems}} · <span class="bad">{{tn "dash.nvlink.problems" (len .Problems)}}</span>{{end}}
· <a href="{{if $.OnAdminListener}}/nvlink{{else}}/admin/nvlink{{end}}">{{t "dash.nvlink.see_racks"}}</a></p>
{{end}}
{{with .Fleet}}
<h2>What the cluster is doing <span class="opt">last {{.Window}}</span></h2>
<h2>{{t "dash.fleet.title"}} <span class="opt">{{t "dash.fleet.last" .Window}}</span></h2>
{{if .Err}}
{{/* Said outright. An empty panel reads as a healthy cluster with nothing
happening, which is the opposite of not knowing. */}}
<p class="note error-row">No history: {{.Err}}</p>
<p class="note error-row">{{t "dash.no_history" .Err}}</p>
{{else}}
<div class="chart-row">
{{template "tschart" dict "t" .Saturation "title" "Pool saturation, by model"
"note" "The worse of queue depth and cache pressure. 1 is the limit measured as healthy, not the point of failure."}}
{{template "tschart" dict "t" .Ready "title" "Replicas ready, by model"
"note" "A model deliberately stopped has no line at all, which is why an on-demand catalogue looks quiet here."}}
{{template "tschart" dict "t" .Saturation "title" (t "dash.fleet.saturation") "note" (t "dash.fleet.saturation_note")}}
{{template "tschart" dict "t" .Ready "title" (t "dash.fleet.ready") "note" (t "dash.fleet.ready_note")}}
</div>
{{if $.AnyGPUs}}
<div class="chart-row">
{{template "tschart" dict "t" .GPUBusy "title" "GPUs busy, by class"
"note" "The average card of each class. Low here with replicas placed is capacity held and not asked for; each node's page has its cards one by one."}}
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory in use, by class"
"note" "Engines reserve most of a card's memory when they start, so this follows replicas placed more than requests served."}}
{{template "tschart" dict "t" .GPUBusy "title" (t "dash.fleet.gpu_busy") "note" (t "dash.fleet.gpu_busy_note")}}
{{template "tschart" dict "t" .GPUMemory "title" (t "dash.fleet.gpu_memory") "note" (t "dash.fleet.gpu_memory_note")}}
</div>
{{end}}
<div class="chart-row">
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica"
"note" "Requests accepted and not yet started. The first thing to move when a replica is falling behind."}}
{{template "tschart" dict "t" .KV "title" "KV cache occupancy, by replica"
"note" "How full the attention cache is. Long contexts fill it with an empty queue, which is why saturation reads both."}}
{{template "tschart" dict "t" .Queue "title" (t "dash.fleet.queue") "note" (t "dash.fleet.queue_note")}}
{{template "tschart" dict "t" .KV "title" (t "dash.fleet.kv") "note" (t "dash.fleet.kv_note")}}
</div>
{{end}}
{{end}}
<hr class="part">
<h2>Traffic <span class="note">— last 24 hours, hourly</span></h2>
<h2>{{t "dash.traffic"}} <span class="note">{{t "dash.traffic.window"}}</span></h2>
{{if .HasCharts}}
<p class="note">Requests</p>{{.RequestChart}}
<p class="note">{{t "dash.traffic.requests"}}</p>{{.RequestChart}}
<div class="chart-row">
<div><p class="note">Tokens</p>{{.TokenChart}}</div>
<div><p class="note">Mean latency</p>{{.LatencyChart}}</div>
<div><p class="note">{{t "dash.traffic.tokens"}}</p>{{.TokenChart}}</div>
<div><p class="note">{{t "dash.traffic.latency"}}</p>{{.LatencyChart}}</div>
</div>
{{else if .AuthEnabled}}
<p class="empty">No traffic recorded yet.</p>
<p class="empty">{{t "dash.traffic.none"}}</p>
{{end}}
{{if .HasPeople}}
<h2>People</h2>
<table><tr><th>Person</th><th class="num">Keys</th><th class="num">Requests</th><th class="num">Tokens</th><th class="num">Errors</th><th class="num">Spend</th><th>Last seen</th></tr>
<h2>{{t "dash.people"}}</h2>
<table><tr><th>{{t "dash.col.person"}}</th><th class="num">{{t "dash.people.keys"}}</th><th class="num">{{t "dash.col.requests"}}</th><th class="num">{{t "dash.col.tokens"}}</th><th class="num">{{t "dash.col.errors"}}</th><th class="num">{{t "dash.col.spend"}}</th><th>{{t "dash.people.last_seen"}}</th></tr>
{{range .Users}}<tr>
<td>{{if .Email}}{{.Email}}{{else}}<span class="note">operator-issued keys</span>{{end}}</td>
<td class="num">{{.Keys}}</td>
<td class="num">{{.Requests}}</td>
<td>{{if .Email}}{{.Email}}{{else}}<span class="note">{{t "dash.people.operator_keys"}}</span>{{end}}</td>
<td class="num">{{count .Keys}}</td>
<td class="num">{{count .Requests}}</td>
<td class="num">{{compact .Tokens}}</td>
<td class="num">{{if .Errors}}<span class="bad">{{.Errors}}</span>{{else}}—{{end}}</td>
<td class="num">{{if .Errors}}<span class="bad">{{count .Errors}}</span>{{else}}—{{end}}</td>
<td class="num">{{units .CostMicros}}</td>
<td>{{ago .LastSeen}}</td>
</tr>{{end}}</table>
{{end}}
{{if .Models2}}
<h2>What is being run</h2>
<table><tr><th>Model</th><th class="num">Requests</th><th class="num">Tokens</th><th class="num">Errors</th><th class="num">Mean latency</th></tr>
<h2>{{t "dash.running"}}</h2>
<table><tr><th>{{t "dash.col.model"}}</th><th class="num">{{t "dash.col.requests"}}</th><th class="num">{{t "dash.col.tokens"}}</th><th class="num">{{t "dash.col.errors"}}</th><th class="num">{{t "dash.traffic.latency"}}</th></tr>
{{range .Models2}}<tr>
<td><code>{{.Model}}</code></td>
<td class="num">{{.Requests}}</td>
<td class="num">{{count .Requests}}</td>
<td class="num">{{compact .Tokens}}</td>
<td class="num">{{if .Errors}}<span class="bad">{{.Errors}}</span>{{else}}—{{end}}</td>
<td class="num">{{if .Errors}}<span class="bad">{{count .Errors}}</span>{{else}}—{{end}}</td>
<td class="num">{{latency .AvgLatency}}</td>
</tr>{{end}}</table>
<p class="note">Counted by the model that actually answered, which differs from the one
requested whenever a fallback served it.</p>
<p class="note">{{t "dash.running.note"}}</p>
{{end}}
<h2>Usage <span class="note">— last 24 hours</span></h2>
<h2>{{t "dash.usage"}} <span class="note">{{t "dash.usage.window"}}</span></h2>
{{if .AuthEnabled}}
{{if .Usage}}
<table><tr><th>Person</th><th>Through</th><th>Spends against</th><th class="num">Requests</th><th class="num">Errors</th><th class="num">Prompt</th><th class="num">Completion</th><th class="num">Spend</th><th>Last</th></tr>
<table><tr><th>{{t "dash.col.person"}}</th><th>{{t "dash.usage.through"}}</th><th>{{t "dash.usage.against"}}</th><th class="num">{{t "dash.col.requests"}}</th><th class="num">{{t "dash.col.errors"}}</th><th class="num">{{t "dash.usage.prompt"}}</th><th class="num">{{t "dash.usage.completion"}}</th><th class="num">{{t "dash.col.spend"}}</th><th>{{t "dash.usage.last"}}</th></tr>
{{range .Usage}}<tr>
<td>{{or .Email "—"}}</td>
<td>{{if .Chat}}chat page{{else if .KeyName}}{{.KeyName}} <code class="note">{{.KeyPrefix}}</code>{{else}}<code title="{{.KeyID}}">{{.KeyID}}</code>{{end}}</td>
<td>{{if .ProjectPath}}{{.ProjectPath}}{{else if .ProjectID}}<code>{{.ProjectID}}</code>{{else}}personal{{end}}</td>
<td class="num">{{.Requests}}</td>
<td class="num">{{if .Errors}}<span class="state-failed">{{.Errors}}</span>{{else}}0{{end}}</td>
<td>{{if .Chat}}{{t "dash.usage.chat_page"}}{{else if .KeyName}}{{.KeyName}} <code class="note">{{.KeyPrefix}}</code>{{else}}<code title="{{.KeyID}}">{{.KeyID}}</code>{{end}}</td>
<td>{{if .ProjectPath}}{{.ProjectPath}}{{else if .ProjectID}}<code>{{.ProjectID}}</code>{{else}}{{t "dash.usage.personal"}}{{end}}</td>
<td class="num">{{count .Requests}}</td>
<td class="num">{{if .Errors}}<span class="state-failed">{{count .Errors}}</span>{{else}}0{{end}}</td>
<td class="num">{{compact .Prompt}}</td>
<td class="num">{{compact .Completion}}</td>
<td class="num">{{units .CostMicros}}</td>
<td><span class="note">{{ago .LastAt}}</span></td>
</tr>{{end}}</table>
<p class="note">Spend is in units, whatever a unit was configured to mean. Usage is
written in batches, so the last few seconds may be missing.</p>
{{else}}<p class="empty">No requests recorded in the last 24 hours.</p>{{end}}
<p class="note">{{t "dash.usage.note"}}</p>
{{else}}<p class="empty">{{t "dash.usage.none"}}</p>{{end}}
{{else}}
<p class="empty">Authentication is off, so there is no identity to account against.</p>
<p class="empty">{{t "dash.usage.auth_off"}}</p>
{{end}}
{{if not .OnAdminListener}}
<p class="note">Metrics, pprof and the management API are on the admin listener,
which is bound to loopback and not reachable from here. pprof can dump the
process heap, where API keys sit in plaintext, which is why that port is not
exposed:
<p class="note">{{t "dash.admin_listener"}}
<code>ssh -L 9090:127.0.0.1:9090 &lt;control-plane&gt;</code></p>
{{end}}
</div>
<footer>
<span>llmgw {{.Version}} · refreshes every {{.RefreshSec}}s</span>
<span>llmgw {{.Version}} · {{t "common.refreshes_every" .RefreshSec}}</span>
<span>{{.Now}}</span>
</footer>
{{template "langs"}}</footer>
</div></body></html>{{end}}
{{/* One chart: the SVG, its legend, and the sentence that says how to read it.
@@ -349,10 +344,10 @@ exposed:
<div class="chart-wrap">
<h3 class="chart-title">{{.title}}</h3>
{{if .t.Empty}}
<p class="empty">Nothing recorded yet.</p>
<p class="empty">{{t "chart.nothing_yet"}}</p>
{{else}}
{{.t.SVG}}
<span class="chart-peak">peak {{.t.Peak}}</span>
<span class="chart-peak">{{t "chart.peak" .t.Peak}}</span>
{{if gt (len .t.Legend) 1}}
<div class="legend">
{{range .t.Legend}}<span><i style="background:var({{cssvar .Colour}})"></i>{{.Name}} <b>{{.Last}}</b></span>{{end}}
+33 -28
View File
@@ -1,35 +1,36 @@
{{define "fabrics"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Fabrics · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "fabric.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap wide">
<header>
<h1>Fabrics</h1>
<p class="sub">{{len .List}} physical fabric{{if ne (len .List) 1}}s{{end}}, as their adapters find them</p>
{{if .OnAdminListener}}<nav><a href="/">cluster</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
<h1>{{t "fabric.title"}}</h1>
<p class="sub">{{tn "fabric.sub" (len .List)}}</p>
{{if .OnAdminListener}}<nav><a href="/">{{t "fabric.nav.cluster"}}</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
</header>
{{if not .List}}
<p class="empty">No InfiniBand or RoCE port has reported yet. A node's adapters appear here with its first
heartbeat, and its fabric's map within five minutes of that.</p>
<p class="empty">{{t "fabric.empty"}}</p>
{{else}}
<nav class="pills" aria-label="Fabric">{{range .List}}<a href="{{$.Base}}/{{.ID}}"{{if $.Current}}{{if eq .ID $.Current.ID}} class="on" aria-current="page"{{end}}{{end}}>
{{if .Declared}}{{range $i, $d := .Declared}}{{if $i}}, {{end}}{{$d}}{{end}}{{else}}undeclared{{end}}
<nav class="pills" aria-label="{{t "fabric.pills"}}">{{range .List}}<a href="{{$.Base}}/{{.ID}}"{{if $.Current}}{{if eq .ID $.Current.ID}} class="on" aria-current="page"{{end}}{{end}}>
{{if .Declared}}{{range $i, $d := .Declared}}{{if $i}}, {{end}}{{$d}}{{end}}{{else}}{{t "fabric.undeclared"}}{{end}}
<span class="note">{{.Label}}</span></a>{{end}}</nav>
{{end}}
{{with .Current}}
<div class="fabric-head">
<div>
<h2>{{if .Declared}}{{range $i, $d := .Declared}}{{if $i}}, {{end}}<code>{{$d}}</code>{{end}}{{else}}undeclared{{end}}</h2>
<h2>{{if .Declared}}{{range $i, $d := .Declared}}{{if $i}}, {{end}}<code>{{$d}}</code>{{end}}{{else}}{{t "fabric.undeclared"}}{{end}}</h2>
<p class="sub">{{.Label}}
{{with .SM}} · master <code>{{.GUID}}</code>{{if .LID}} LID {{.LID}}{{end}}{{end}}
{{if .SMs}} · {{len .SMs}} subnet manager{{if gt (len .SMs) 1}}s{{end}}{{end}}
{{if not .ScannedAt.IsZero}} · mapped {{ago .ScannedAt}} through {{.ScannedBy}} in {{.TookMs}} ms{{end}}</p>
{{with .SM}} · {{t "fabric.master"}} <code>{{.GUID}}</code>{{if .LID}} {{t "fabric.lid" .LID}}{{end}}{{end}}
{{if .SMs}} · {{tn "fabric.sms" (len .SMs)}}{{end}}
{{if not .ScannedAt.IsZero}} · {{t "fabric.mapped" (ago .ScannedAt) .ScannedBy (count .TookMs)}}{{end}}</p>
</div>
</div>
{{if .Problems}}<ul class="fabric-problems">{{range .Problems}}
<li class="{{.Level}}"><span class="tag{{if eq (printf "%s" .Level) "fail"}} bad{{end}}">{{.Level}}</span> {{.Message}}</li>{{end}}
{{/* The level is a class and a word; the message is the fabric
checker's own, in English. */}}
<li class="{{.Level}}"><span class="tag{{if eq (printf "%s" .Level) "fail"}} bad{{end}}">{{t (print "fabric.level." .Level)}}</span> {{.Message}}</li>{{end}}
</ul>{{end}}
<div id="fabric-map" class="fabric-map" data-src="{{$.Base}}/{{.ID}}/topology.json"
@@ -38,32 +39,36 @@ heartbeat, and its fabric's map within five minutes of that.</p>
<div class="fm-stats"></div>
<div class="fm-tools">
<span class="pills small">
<button type="button" class="on" data-mode="speed">Speed</button>
<button type="button" data-mode="rails">Rails</button>
<button type="button" class="on" data-mode="speed">{{t "fabric.mode.speed"}}</button>
<button type="button" data-mode="rails">{{t "fabric.mode.rails"}}</button>
</span>
<button type="button" class="fm-toggle" data-toggle="traffic">Traffic</button>
<button type="button" class="fm-toggle" data-toggle="problems">Only problems</button>
<input type="search" class="fm-search" placeholder="Find a node" aria-label="Find a node">
<button type="button" class="fm-toggle" data-reset title="Or double-click the map">Reset view</button>
<button type="button" class="fm-toggle" data-toggle="traffic">{{t "fabric.traffic"}}</button>
<button type="button" class="fm-toggle" data-toggle="problems">{{t "fabric.only_problems"}}</button>
<input type="search" class="fm-search" placeholder="{{t "fabric.find"}}" aria-label="{{t "fabric.find"}}">
<button type="button" class="fm-toggle" data-reset title="{{t "fabric.reset_hint"}}">{{t "fabric.reset"}}</button>
</div>
</div>
<div class="fm-stage">
<div class="fm-canvas"><noscript><p class="fm-empty">The map is drawn in the browser; the problems above are the same
without it.</p></noscript></div>
{{/* The paragraph is outside the noscript, which holds only its words:
the drawing empties the canvas before it draws, so with script it
is gone at once. */}}
<div class="fm-canvas"><p class="fm-empty"><noscript>{{t "fabric.noscript"}}</noscript></p></div>
<div class="fm-tip" hidden></div>
<aside class="fm-panel" hidden></aside>
</div>
<div class="fm-legend">
<span><i class="sw sp-xdr"></i>XDR</span><span><i class="sw sp-ndr"></i>NDR</span><span><i class="sw sp-hdr"></i>HDR</span>
<span><i class="sw sp-edr"></i>EDR</span><span><i class="sw sp-slow"></i>slower</span>
<span><i class="sw degraded"></i>below its kind</span>
<span><i class="dot known"></i>this cluster's adapter</span><span><i class="dot"></i>another host</span><span><i class="dot down"></i>port down</span>
<span class="note">scroll to zoom · drag to move · click for detail</span>
{{/* InfiniBand's names for its speeds, the same in every language. */}}
<span translate="no"><i class="sw sp-xdr"></i>XDR</span><span translate="no"><i class="sw sp-ndr"></i>NDR</span><span translate="no"><i class="sw sp-hdr"></i>HDR</span>
<span translate="no"><i class="sw sp-edr"></i>EDR</span><span><i class="sw sp-slow"></i>{{t "fabric.legend.slower"}}</span>
<span><i class="sw degraded"></i>{{t "fabric.legend.below"}}</span>
<span><i class="dot known"></i>{{t "fabric.legend.ours"}}</span><span><i class="dot"></i>{{t "fabric.legend.other"}}</span><span><i class="dot down"></i>{{t "fabric.legend.down"}}</span>
<span class="note">{{t "fabric.legend.how"}}</span>
</div>
</div>
<script type="application/json" id="fabric-data">{{$.Data}}</script>
<script type="application/json" id="fabric-i18n">{{$.Strings}}</script>
<script src="{{asset "fabric.js"}}" defer></script>
{{end}}
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}
+9
View File
@@ -0,0 +1,9 @@
{{/* The language switcher at the foot of every page: each language by its
own name, so someone looking for Türkçe finds that word, and the one in
use marked rather than offered. A plain link, so it works without a
script; /_lang remembers the choice and comes back here. */}}
{{define "langs"}}<nav class="langs" aria-label="{{t "lang.switch"}}">
{{- range $i, $l := languages}}{{if $i}} · {{end}}
{{- if eq $l.Code lang}}<b lang="{{$l.Code}}">{{$l.Name}}</b>
{{- else}}<a href="/_lang/{{$l.Code}}" hreflang="{{$l.Code}}" lang="{{$l.Code}}">{{$l.Name}}</a>{{end}}
{{- end}}</nav>{{end}}
+43 -56
View File
@@ -1,27 +1,25 @@
{{define "landing"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>llmgw</title>
<meta name="description" content="One OpenAI-compatible endpoint in front of self-hosted models.">
<meta name="description" content="{{t "landing.description"}}">
<link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
{{if .Notice}}<p class="banner note">{{.Notice}}</p>{{end}}
{{if not .AuthEnabled}}
<p class="banner warn-banner">This gateway has no API keys configured, so it serves anyone
who can reach it. That is a deployment choice, and only safe on a trusted network.</p>
<p class="banner warn-banner">{{t "landing.open_warning"}}</p>
{{end}}
<header class="hero">
<div class="hero-text">
<h1>llmgw</h1>
<p class="sub">One OpenAI-compatible endpoint in front of your own GPUs.</p>
<p class="lede">Point any OpenAI client at this address. The API is the one you
already have a client for — only the base URL changes.</p>
<p class="sub">{{t "landing.sub"}}</p>
<p class="lede">{{t "landing.lede"}}</p>
<p class="cta">
{{if .SignupOpen}}<a class="btn" href="/signup">Create an account</a>
<a class="btn ghost" href="/login">Sign in</a>
{{else if .AuthEnabled}}<a class="btn" href="/login">Sign in</a>{{end}}
{{if .SignupOpen}}<a class="btn" href="/signup">{{t "landing.create_account"}}</a>
<a class="btn ghost" href="/login">{{t "landing.sign_in"}}</a>
{{else if .AuthEnabled}}<a class="btn" href="/login">{{t "landing.sign_in"}}</a>{{end}}
</p>
</div>
@@ -30,7 +28,7 @@ who can reach it. That is a deployment choice, and only safe on a trusted networ
the address. */}}
<figure class="schematic" aria-labelledby="schematic-cap">
<svg viewBox="0 0 420 200" role="img"
aria-label="Clients reach one endpoint, which places work on GPU nodes across separate fabrics.">
aria-label="{{t "landing.schematic"}}">
<defs>
<marker id="a" viewBox="0 0 8 8" refX="7" refY="4" markerWidth="6" markerHeight="6" orient="auto">
<path d="M0 0 L8 4 L0 8 z" fill="var(--viz-grid)"/>
@@ -40,11 +38,11 @@ who can reach it. That is a deployment choice, and only safe on a trusted networ
<!-- clients -->
<g font-size="10" fill="var(--muted)">
<rect x="2" y="42" width="84" height="26" rx="6" fill="var(--card)" stroke="var(--line)"/>
<text x="44" y="59" text-anchor="middle" fill="var(--fg)">your client</text>
<text x="44" y="59" text-anchor="middle" fill="var(--fg)">{{t "landing.schematic.client"}}</text>
<rect x="2" y="86" width="84" height="26" rx="6" fill="var(--card)" stroke="var(--line)"/>
<text x="44" y="103" text-anchor="middle" fill="var(--fg)">editor agent</text>
<text x="44" y="103" text-anchor="middle" fill="var(--fg)">{{t "landing.schematic.editor"}}</text>
<rect x="2" y="130" width="84" height="26" rx="6" fill="var(--card)" stroke="var(--line)"/>
<text x="44" y="147" text-anchor="middle" fill="var(--fg)">a script</text>
<text x="44" y="147" text-anchor="middle" fill="var(--fg)">{{t "landing.schematic.script"}}</text>
</g>
<g fill="none" stroke="var(--viz-grid)" stroke-width="2" marker-end="url(#a)">
<path d="M88 55 C 118 55 118 92 146 92"/>
@@ -56,9 +54,9 @@ who can reach it. That is a deployment choice, and only safe on a trusted networ
<rect x="150" y="70" width="72" height="60" rx="8"
fill="var(--accent)" opacity=".12" stroke="var(--accent)" stroke-width="1.5"/>
<text x="186" y="96" text-anchor="middle" font-size="12" font-weight="600"
fill="var(--fg)">llmgw</text>
fill="var(--fg)" translate="no">llmgw</text>
<text x="186" y="112" text-anchor="middle" font-size="9"
fill="var(--muted)">one address</text>
fill="var(--muted)">{{t "landing.schematic.address"}}</text>
<g fill="none" stroke="var(--viz-grid)" stroke-width="2" marker-end="url(#a)">
<path d="M224 88 C 254 88 254 46 286 46"/>
@@ -69,77 +67,66 @@ who can reach it. That is a deployment choice, and only safe on a trusted networ
<g font-size="9" fill="var(--muted)">
<rect x="288" y="18" width="128" height="58" rx="8"
fill="none" stroke="var(--line)" stroke-dasharray="4 3"/>
<text x="296" y="32">fabric</text>
<text x="296" y="32">{{t "landing.schematic.fabric"}}</text>
<rect x="296" y="40" width="26" height="26" rx="4" fill="var(--viz-1)" opacity=".85"/>
<rect x="330" y="40" width="26" height="26" rx="4" fill="var(--viz-1)" opacity=".85"/>
<rect x="364" y="40" width="26" height="26" rx="4" fill="var(--viz-1)" opacity=".85"/>
<rect x="288" y="118" width="128" height="58" rx="8"
fill="none" stroke="var(--line)" stroke-dasharray="4 3"/>
<text x="296" y="132">fabric</text>
<text x="296" y="132">{{t "landing.schematic.fabric"}}</text>
<rect x="296" y="140" width="26" height="26" rx="4" fill="var(--viz-3)" opacity=".85"/>
<rect x="330" y="140" width="26" height="26" rx="4" fill="var(--viz-3)" opacity=".85"/>
<rect x="364" y="140" width="26" height="26" rx="4" fill="var(--viz-3)" opacity=".85"/>
</g>
</svg>
<figcaption id="schematic-cap">Clients see one address. Work is placed on GPUs that
can actually run it — and a model split across machines never straddles two fabrics.</figcaption>
<figcaption id="schematic-cap">{{t "landing.schematic_caption"}}</figcaption>
</figure>
</header>
<section class="url-card">
<span class="k">Base URL</span>
<span class="k">{{t "landing.base_url"}}</span>
<code>{{.BaseURL}}/v1</code>
</section>
<h2>Three steps</h2>
<h2>{{t "landing.three_steps"}}</h2>
<ol class="steps">
<li>
<span class="n">1</span>
<h3>Get a key</h3>
<h3>{{t "landing.step1"}}</h3>
{{if .SignupOpen}}
<p><a href="/signup">Create an account</a>. Someone approves it, and then you issue
your own keys from your <a href="/profile">profile</a> — as many as you need,
revoking them yourself when a laptop is retired or a script is done.</p>
<p>{{th "landing.step1_signup"}}</p>
{{else if .Contact}}
<p>Ask {{if .ContactHref}}<a href="{{.ContactHref}}">{{.Contact}}</a>{{else}}<strong>{{.Contact}}</strong>{{end}}
for one.{{if .AuthEnabled}} If you already have an account,
<a href="/login">sign in</a>.{{end}}</p>
<p>{{th "landing.step1_contact" .ContactHTML}}{{if .AuthEnabled}}{{th "landing.step1_contact_signin"}}{{end}}</p>
{{else}}
<p>Keys are issued by whoever operates this gateway. If you do not know who that
is, the address you reached this page at is the place to start asking.</p>
<p>{{t "landing.step1_nobody"}}</p>
{{end}}
<p class="note">A key is shown once and never again — only its hash is kept here —
so put it in a password manager straight away.</p>
<p class="note">{{t "landing.step1_shown_once"}}</p>
</li>
<li>
<span class="n">2</span>
<h3>Point your client at it</h3>
<p>Set the base URL above and your key. Every OpenAI SDK, and every tool built on
one, takes both as configuration.</p>
<h3>{{t "landing.step2"}}</h3>
<p>{{t "landing.step2_text"}}</p>
<pre><code>export OPENAI_BASE_URL={{.BaseURL}}/v1
export OPENAI_API_KEY=$LLMGW_API_KEY</code></pre>
</li>
<li>
<span class="n">3</span>
<h3>Pick a model</h3>
<h3>{{t "landing.step3"}}</h3>
{{if .AuthEnabled}}
<p><code>GET /v1/models</code> lists what is available to your key. It needs the
key, which is why this page does not list them: this page is served to anyone who
can reach the address.</p>
<p>{{th "landing.step3_keyed"}}</p>
{{else}}
<p><code>GET /v1/models</code> lists what is available.</p>
<p>{{th "landing.step3_open"}}</p>
{{end}}
<p class="note">Sending the placeholder <code>&lt;model-name&gt;</code> unchanged is
answered with the list of models you may use, rather than with an error.</p>
<p class="note">{{th "landing.step3_placeholder"}}</p>
</li>
</ol>
<h2>Connect your client</h2>
<h2>{{t "landing.connect"}}</h2>
<div class="tabs">
<input type="radio" name="client" id="tab-curl" checked><label for="tab-curl">curl</label>
<input type="radio" name="client" id="tab-py"><label for="tab-py">Python</label>
<input type="radio" name="client" id="tab-oc"><label for="tab-oc">OpenCode</label>
<input type="radio" name="client" id="tab-curl" checked><label for="tab-curl" translate="no">curl</label>
<input type="radio" name="client" id="tab-py"><label for="tab-py" translate="no">Python</label>
<input type="radio" name="client" id="tab-oc"><label for="tab-oc" translate="no">OpenCode</label>
<div class="panels">
<pre class="panel"><code>curl {{.BaseURL}}/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
@@ -176,22 +163,22 @@ print(reply.choices[0].message.content)</code></pre>
}</code></pre>
</div>
</div>
<p class="note">Anything that speaks the chat-completions API works, whether or not it
is listed here — there is nothing llmgw-specific to install.</p>
<p class="note">{{t "landing.anything_works"}}</p>
<h2>Once you have a key</h2>
<h2>{{t "landing.once_you_have_key"}}</h2>
<ul class="endpoints">
<li><code>GET {{.BaseURL}}/v1/models</code><span>what your key may use</span></li>
<li><code>GET {{.BaseURL}}/me</code><span>its limits, and what it has spent</span></li>
<li><code>GET {{.BaseURL}}/v1/models</code><span>{{t "landing.models_endpoint"}}</span></li>
<li><code>GET {{.BaseURL}}/me</code><span>{{t "landing.me_endpoint"}}</span></li>
</ul>
<footer><span><a href="/about">llmgw {{.Version}}</a></span><span>{{.Now}}</span></footer>
<footer><span><a href="/about">llmgw {{.Version}}</a></span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div>
<script>
// Vanilla and inline, for the same reason this page has no build step. It is an
// enhancement: the button only exists where the clipboard API does, so nothing
// here can leave a dead control on the page.
const copyWord = {{t "common.copy"}}, copiedWord = {{t "common.copied"}};
if (navigator.clipboard) {
for (const pre of document.querySelectorAll('pre')) {
// Read the text before the button joins the element, or "copy" ends up in
@@ -200,11 +187,11 @@ if (navigator.clipboard) {
const b = document.createElement('button');
b.type = 'button';
b.className = 'copy';
b.textContent = 'copy';
b.textContent = copyWord;
b.addEventListener('click', () => {
navigator.clipboard.writeText(text).then(() => {
b.textContent = 'copied';
setTimeout(() => { b.textContent = 'copy'; }, 1200);
b.textContent = copiedWord;
setTimeout(() => { b.textContent = copyWord; }, 1200);
});
});
pre.appendChild(b);
+11 -11
View File
@@ -1,33 +1,33 @@
{{/* The one menu every signed-in page draws. The order never changes and the
page you are on is marked, not offered, so moving between pages reads as
moving within one program. */}}
{{define "nav"}}<nav class="menu" aria-label="Main">
{{define "nav"}}<nav class="menu" aria-label="{{t "nav.aria"}}">
{{- if and .Org (not .Admin)}}
<a href="/org"{{if eq .Current "org"}} aria-current="page"{{end}}>Organization</a>
<a href="/org"{{if eq .Current "org"}} aria-current="page"{{end}}>{{t "nav.org"}}</a>
<span class="menu-gap" aria-hidden="true"></span>
{{- end}}
{{- if and .Admin (eq .Role "billing")}}
<a href="/admin/usage"{{if eq .Current "usage"}} aria-current="page"{{end}}>Usage</a>
<a href="/admin/quotas"{{if eq .Current "quotas"}} aria-current="page"{{end}}>Quotas</a>
<a href="/admin/usage"{{if eq .Current "usage"}} aria-current="page"{{end}}>{{t "nav.usage"}}</a>
<a href="/admin/quotas"{{if eq .Current "quotas"}} aria-current="page"{{end}}>{{t "nav.quotas"}}</a>
<span class="menu-gap" aria-hidden="true"></span>
{{- else if .Admin}}
<a href="/admin"{{if eq .Current "cluster"}} aria-current="page"{{end}}>Cluster</a>
<a href="/admin"{{if eq .Current "cluster"}} aria-current="page"{{end}}>{{t "nav.cluster"}}</a>
{{- /* Limits and usage are tabs of the people page, not pages of their own. */}}
<a href="/admin/users"{{if eq .Current "people"}} aria-current="page"{{end}}>People</a>
<a href="/admin/users"{{if eq .Current "people"}} aria-current="page"{{end}}>{{t "nav.people"}}</a>
{{- if .Mirror}}
<a href="/admin/mirror"{{if eq .Current "mirror"}} aria-current="page"{{end}}>Mirror</a>
<a href="/admin/mirror"{{if eq .Current "mirror"}} aria-current="page"{{end}}>{{t "nav.mirror"}}</a>
{{- end}}
{{- if .Guard}}
<a href="/admin/guard"{{if eq .Current "guard"}} aria-current="page"{{end}}>Guardrails</a>
<a href="/admin/guard"{{if eq .Current "guard"}} aria-current="page"{{end}}>{{t "nav.guard"}}</a>
{{- end}}
{{- /* An organization is a panel of the people page for an administrator;
/org is the managers' view, and leads an administrator there. */}}
<span class="menu-gap" aria-hidden="true"></span>
{{- end}}
<a href="/chat"{{if eq .Current "chat"}} aria-current="page"{{end}}>Chat</a>
<a href="/profile"{{if eq .Current "keys"}} aria-current="page"{{end}}>Account</a>
<a href="/chat"{{if eq .Current "chat"}} aria-current="page"{{end}}>{{t "nav.chat"}}</a>
<a href="/profile"{{if eq .Current "keys"}} aria-current="page"{{end}}>{{t "nav.account"}}</a>
<form method="post" action="/logout" class="inline">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<button type="submit" class="link">Sign out</button>
<button type="submit" class="link">{{t "nav.sign_out"}}</button>
</form>
</nav>{{end}}
+65 -63
View File
@@ -1,7 +1,7 @@
{{define "node"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<noscript><meta http-equiv="refresh" content="{{.RefreshSec}}"></noscript>
<noscript>{{.RefreshMeta}}</noscript>
<meta name="htmx-config" content='{"allowEval":false}'>
<script src="/_static/htmx-2.0.11.min.js"></script>
<script defer src="/_static/pages.js"></script>
@@ -9,10 +9,10 @@
</head><body><div class="wrap">
<header>
<h1>{{.Node.ID}}</h1>
<p class="sub"><span class="state state-{{.Node.State}}">{{.Node.State}}</span>
<p class="sub"><span class="state state-{{.Node.State}}">{{t (print "dash.state." .Node.State)}}</span>
{{with .Node.OS}} · {{.}}{{end}}{{with .Node.MgmtAddr}} · {{.}}{{end}}
· last seen {{or .Node.LastSeenAgo "never"}} ago · <a href="{{.Logs}}">logs</a></p>
{{if .OnAdminListener}}<nav><a href="{{.Back}}">cluster</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
· {{if .Node.LastSeenAgo}}{{t "node.last_seen" .Node.LastSeenAgo}}{{else}}{{t "node.never_seen"}}{{end}} · <a href="{{.Logs}}">{{t "dash.nodes.logs"}}</a></p>
{{if .OnAdminListener}}<nav><a href="{{.Back}}">{{t "node.nav.cluster"}}</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
</header>
{{/* Refreshed in place, like the cluster page. */}}
@@ -20,131 +20,133 @@
{{with .Node}}
<div class="tiles">
{{with .Load}}
<div class="tile"><div class="v">{{printf "%.0f" .CPUPercent}}%</div><div class="k">CPU{{if .CPUs}} · {{.CPUs}} cores{{end}}</div>{{template "minibar" pctbar .CPUPercent}}</div>
<div class="tile"><div class="v">{{bytes .MemUsed}}</div><div class="k">RAM in use · of {{bytes .MemTotal}}</div>{{template "minibar" pctbar .MemPercent}}</div>
<div class="tile"><div class="v">{{printf "%.2f" .Load1}}</div><div class="k">load · {{printf "%.2f" .Load5}} 5m · {{printf "%.2f" .Load15}} 15m</div></div>
<div class="tile"><div class="v">{{num .CPUPercent 0}}%</div><div class="k">CPU{{if .CPUs}} · {{t "node.cores" .CPUs}}{{end}}</div>{{template "minibar" pctbar .CPUPercent}}</div>
<div class="tile"><div class="v">{{bytes .MemUsed}}</div><div class="k">{{t "node.ram_in_use_of" (bytes .MemTotal)}}</div>{{template "minibar" pctbar .MemPercent}}</div>
<div class="tile"><div class="v">{{num .Load1 2}}</div><div class="k">{{t "node.load" (num .Load5 2) (num .Load15 2)}}</div></div>
{{/* Only what this machine has sensors for: a VM has none, and a tile
saying 0 °C would be a reading nobody took. */}}
{{if or .CPUTempC .CPUPowerW}}<div class="tile"><div class="v">{{if .CPUTempC}}{{printf "%.0f" .CPUTempC}} °C{{else}}—{{end}}</div><div class="k">CPU{{if .CPUPowerW}} · {{printf "%.0f" .CPUPowerW}} W{{end}}</div></div>{{end}}
{{if .InletTempC}}<div class="tile"><div class="v">{{printf "%.1f" .InletTempC}} °C</div><div class="k">inlet air</div></div>{{end}}
{{if or .CPUTempC .CPUPowerW}}<div class="tile"><div class="v">{{if .CPUTempC}}{{num .CPUTempC 0}} °C{{else}}—{{end}}</div><div class="k">CPU{{if .CPUPowerW}} · {{num .CPUPowerW 0}} W{{end}}</div></div>{{end}}
{{if .InletTempC}}<div class="tile"><div class="v">{{num .InletTempC 1}} °C</div><div class="k">{{t "node.inlet"}}</div></div>{{end}}
{{else}}
<div class="tile"><div class="v">—</div><div class="k">CPU and RAM not measured lately</div></div>
<div class="tile"><div class="v">—</div><div class="k">{{t "node.not_measured"}}</div></div>
{{end}}
{{if .DiskTotalGB}}<div class="tile"><div class="v">{{.DiskFreeGB}}G</div><div class="k">disk free · of {{.DiskTotalGB}}G</div></div>{{end}}
{{if .GPUTotal}}<div class="tile"><div class="v">{{.GPUFree}}/{{.GPUTotal}}</div><div class="k">GPUs free</div></div>{{end}}
{{if .DiskTotalGB}}<div class="tile"><div class="v">{{.DiskFreeGB}}G</div><div class="k">{{t "node.disk_free" .DiskTotalGB}}</div></div>{{end}}
{{if .GPUTotal}}<div class="tile"><div class="v">{{.GPUFree}}/{{.GPUTotal}}</div><div class="k">{{t "node.gpus_free"}}</div></div>{{end}}
</div>
{{/* The node's warnings and what is unhealthy come from the control plane
and the agent, in English. */}}
{{if .Warnings}}{{range .Warnings}}<p class="note bad">{{.}}</p>{{end}}{{end}}
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
{{with .Load}}{{if .GPUs}}
<h2>GPUs</h2>
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by, and wear</th></tr>
<h2>{{t "node.gpus"}}</h2>
<table class="rows"><tr><th class="num">GPU</th><th>{{t "node.gpu.busy"}}</th><th>{{t "node.gpu.memory_used"}}</th><th class="num">{{t "node.gpu.temp"}}</th><th class="num">{{t "node.gpu.memory_temp"}}</th><th>{{t "node.gpu.power"}}</th><th class="num">{{t "node.gpu.clock"}}</th><th>{{t "node.gpu.held_back"}}</th></tr>
{{range .GPUs}}<tr>
<td class="num">{{.Index}}</td>
{{/* From the last sample; a dash where the driver or an older agent did
not say, rather than a zero nobody measured. */}}
<td class="load">{{if and .MemoryTotalMB (ge .UtilPercent 0.0)}}{{template "minibar" pctbar .UtilPercent}}<span class="load-v">{{printf "%.0f" .UtilPercent}}%{{if ge .MemUtilPercent 0.0}} <span class="note">memory {{printf "%.0f" .MemUtilPercent}}%</span>{{end}}</span>{{else}}—{{end}}</td>
<td class="load">{{if .MemoryTotalMB}}{{template "minibar" pctbar (percent .MemoryUsedMB .MemoryTotalMB)}}<span class="load-v">{{printf "%.1f" (gib .MemoryUsedMB)}} <span class="note">of {{printf "%.0f" (gib .MemoryTotalMB)}} GB</span></span>{{else}}—{{end}}</td>
<td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
<td class="num">{{if .SMClockMHz}}{{printf "%.0f" .SMClockMHz}} MHz{{else}}—{{end}}</td>
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{.PCIeReplays}} PCIe retries</span>{{end}}{{end}}</td>
<td class="load">{{if and .MemoryTotalMB (ge .UtilPercent 0.0)}}{{template "minibar" pctbar .UtilPercent}}<span class="load-v">{{num .UtilPercent 0}}%{{if ge .MemUtilPercent 0.0}} <span class="note">{{t "node.gpu.memory_util" (num .MemUtilPercent 0)}}</span>{{end}}</span>{{else}}—{{end}}</td>
<td class="load">{{if .MemoryTotalMB}}{{template "minibar" pctbar (percent .MemoryUsedMB .MemoryTotalMB)}}<span class="load-v">{{num (gib .MemoryUsedMB) 1}} <span class="note">{{t "node.of_gb" (num (gib .MemoryTotalMB) 0)}}</span></span>{{else}}—{{end}}</td>
<td class="num">{{if .TempC}}{{num .TempC 0}} °C{{else}}—{{end}}</td>
<td class="num">{{if .MemoryTempC}}{{num .MemoryTempC 0}} °C{{else}}—{{end}}</td>
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{num .PowerW 0}} W <span class="note">{{t "node.of" (num .PowerLimitW 0)}}</span></span>{{else if .PowerW}}{{num .PowerW 0}} W{{else}}—{{end}}</td>
<td class="num">{{if .SMClockMHz}}{{num .SMClockMHz 0}} MHz{{else}}—{{end}}</td>
{{/* The driver's reasons and the wear are named as the driver and the
inventory name them, in English. */}}
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">{{t "node.gpu.nothing"}}</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{t "node.gpu.ecc" (num .ECCUncorrected 0)}}</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{t "node.gpu.pcie" (count .PCIeReplays)}}</span>{{end}}{{end}}</td>
</tr>{{end}}</table>
{{end}}{{end}}
{{end}}
{{if or .Node.IBPorts .Node.Fabrics}}
<h2>Fabric</h2>
<p class="sub">{{if .Node.Fabrics}}On {{join .Node.Fabrics}} · <a href="{{if $.OnAdminListener}}/fabrics{{else}}/admin/fabrics{{end}}">the map →</a>{{else}}<span class="bad">On no fabric: no port a declared fabric names can carry traffic, so no multi-node replica is placed here.</span>{{end}}</p>
<h2>{{t "node.fabric"}}</h2>
<p class="sub">{{if .Node.Fabrics}}{{t "node.fabric.on" (join .Node.Fabrics)}} · <a href="{{if $.OnAdminListener}}/fabrics{{else}}/admin/fabrics{{end}}">{{t "node.fabric.map"}}</a>{{else}}<span class="bad">{{t "node.fabric.none"}}</span>{{end}}</p>
{{if .Node.IBPorts}}
<table class="rows"><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Physically on</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
<table class="rows"><tr><th>{{t "node.ib.adapter"}}</th><th>{{t "dash.col.state"}}</th><th class="num">{{t "node.ib.rate"}}</th><th>{{t "node.fabric"}}</th><th>{{t "node.ib.physically"}}</th><th>{{t "node.ib.errors"}}</th><th class="num">{{t "node.ib.sent"}}</th><th class="num">{{t "node.ib.received"}}</th></tr>
{{range .Node.IBPorts}}<tr>
<td><code>{{.HCA}}/{{.Port}}</code>{{if eq .LinkLayer "Ethernet"}} <span class="tag">RoCE</span>{{end}}</td>
<td><code>{{.HCA}}/{{.Port}}</code>{{if eq .LinkLayer "Ethernet"}} <span class="tag" translate="no">RoCE</span>{{end}}</td>
{{/* The port's state and its counters are the adapter's own words. */}}
<td class="{{if .Active}}state-ready{{else}}bad{{end}}">{{.State}}{{with .PhysState}} <span class="note">{{.}}</span>{{end}}</td>
<td class="num">{{if .RateGbps}}{{.RateGbps}} Gb/s{{else}}—{{end}}</td>
<td class="num">{{if .RateGbps}}{{t "fabric.gbps" .RateGbps}}{{else}}—{{end}}</td>
<td>{{or .Fabric "—"}}</td>
<td>{{if .Physical}}{{.Physical}}{{else if .SMError}}<span class="note" title="{{.SMError}}">unknown</span>{{else}}—{{end}}</td>
<td>{{if .Errors}}{{range $k, $v := .Errors}}<span class="tag bad">{{$k}} {{$v}}</span> {{end}}{{else}}<span class="note">none</span>{{end}}</td>
<td>{{if .Physical}}{{.Physical}}{{else if .SMError}}<span class="note" title="{{.SMError}}">{{t "node.unknown"}}</span>{{else}}—{{end}}</td>
<td>{{if .Errors}}{{range $k, $v := .Errors}}<span class="tag bad">{{$k}} {{$v}}</span> {{end}}{{else}}<span class="note">{{t "node.none"}}</span>{{end}}</td>
<td class="num">{{if .TxBytes}}{{bytes (u64 .TxBytes)}}{{else}}—{{end}}</td>
<td class="num">{{if .RxBytes}}{{bytes (u64 .RxBytes)}}{{else}}—{{end}}</td>
</tr>{{end}}</table>
<p class="note">Re-read on every heartbeat. A port that goes down takes its fabric off
this node within seconds; one that comes back returns it. Errors are cumulative since
the adapter came up: a number that grows is the one to look at.</p>
<p class="note">{{t "node.ib.note"}}</p>
{{end}}
{{end}}
{{if .Node.GPUs}}
<h2>GPUs and NVLink</h2>
<h2>{{t "node.nvl.title"}}</h2>
<p class="sub">
{{with .Node.NVLinkDomain}}In NVLink domain <code>{{.}}</code>: tensor parallelism can span this node and the rest of its domain.
{{else}}NVLink ends at this machine.{{end}}
{{if or .Node.NVLinkDomain .Node.IMEX}} · IMEX {{if eq .Node.IMEX "active"}}<span class="state-ready">active</span>, {{.Node.IMEXChannels}} channel{{if ne .Node.IMEXChannels 1}}s{{end}}{{else}}<span class="bad">{{or .Node.IMEX "not installed"}}</span>{{end}}{{end}}
{{with .Node.NVLinkDomain}}{{th "node.nvl.domain" .}}
{{else}}{{t "node.nvl.ends_here"}}{{end}}
{{if or .Node.NVLinkDomain .Node.IMEX}} · <span translate="no">IMEX</span> {{if eq .Node.IMEX "active"}}<span class="state-ready">{{t "node.imex.active"}}</span>, {{tn "node.imex.channels" .Node.IMEXChannels}}{{else}}<span class="bad">{{with .Node.IMEX}}{{.}}{{else}}{{t "node.imex.not_installed"}}{{end}}</span>{{end}}{{end}}
</p>
<div class="gpu-wiring">
<table class="rows nvl-matrix" aria-label="NVLink between GPUs">
<table class="rows nvl-matrix" aria-label="{{t "node.nvl.matrix"}}">
<tr><th></th>{{range .Node.GPUs}}<th>{{.Index}}</th>{{end}}</tr>
{{range $a := .Node.GPUs}}<tr><th>GPU {{$a.Index}}</th>
{{range $b := $.Node.GPUs}}{{if eq $a.Index $b.Index}}<td class="self"></td>{{else if hasInt $a.NVLinkPeers $b.Index}}<td class="nv" title="GPU {{$a.Index}} ↔ GPU {{$b.Index}} over NVLink">NV</td>{{else}}<td class="pcie" title="GPU {{$a.Index}} ↔ GPU {{$b.Index}} over PCIe">·</td>{{end}}{{end}}
{{range $b := $.Node.GPUs}}{{if eq $a.Index $b.Index}}<td class="self"></td>{{else if hasInt $a.NVLinkPeers $b.Index}}<td class="nv" title="{{t "node.nvl.over_nvlink" $a.Index $b.Index}}" translate="no">NV</td>{{else}}<td class="pcie" title="{{t "node.nvl.over_pcie" $a.Index $b.Index}}">·</td>{{end}}{{end}}
</tr>{{end}}
</table>
<table class="rows compact gpu-list">
<tr><th>GPU</th><th>Name</th><th class="num">Compute</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
<tr><th>GPU</th><th>{{t "node.gpu.name"}}</th><th class="num">{{t "node.gpu.compute"}}</th><th class="num">{{t "node.gpu.numa"}}</th><th>{{t "node.gpu.nearest"}}</th><th>{{t "node.gpu.multinode"}}</th></tr>
{{range .Node.GPUs}}<tr>
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.ComputeCapability}}</td><td class="num">{{.NUMA}}</td>
<td>{{if .NearestNIC}}<code>{{.NearestNIC}}</code> <span class="note">{{.NICPath}}</span>{{else}}—{{end}}</td>
<td>{{if .Domain}}<span class="state-ready">joined</span>{{if and .Bandwidth (ne .Bandwidth "Full")}} <span class="bad">{{.Bandwidth}} bandwidth</span>{{end}}{{else if .FabricState}}<span class="bad">{{.FabricState}}</span>{{else}}<span class="note">—</span>{{end}}</td>
{{/* The bandwidth and fabric state are the driver's words. */}}
<td>{{if .Domain}}<span class="state-ready">{{t "node.gpu.joined"}}</span>{{if and .Bandwidth (ne .Bandwidth "Full")}} <span class="bad">{{t "node.gpu.bandwidth" .Bandwidth}}</span>{{end}}{{else if .FabricState}}<span class="bad">{{.FabricState}}</span>{{else}}<span class="note">—</span>{{end}}</td>
</tr>{{end}}
</table>
</div>
<p class="note">NV is a pair of cards joined by NVLink; a tensor-parallel group kept to NV pairs runs its
all-reduces at NVLink speed, and placement picks it that way. The nearest adapter is where a card's share of
traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS is across the processors.</p>
<p class="note">{{t "node.nvl.note"}}</p>
{{end}}
<h2>What runs here</h2>
<h2>{{t "node.runs"}}</h2>
{{if .Engines}}
<table class="rows"><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>
<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th>{{t "dash.col.replica"}}</th><th>{{t "dash.col.state"}}</th><th>CPU</th><th class="num">{{t "node.memory"}}</th></tr>
{{range .Engines}}<tr>
<td>{{.Model}}</td>
<td><code>{{.Replica}}</code></td>
<td class="state state-{{.State}}">{{.State}}</td>
<td class="load">{{template "minibar" pctbar .CPUShare}}<span class="load-v">{{printf "%.1f" .CPUCores}} cores</span></td>
<td class="state state-{{.State}}">{{t (print "dash.state." .State)}}</td>
<td class="load">{{template "minibar" pctbar .CPUShare}}<span class="load-v">{{t "node.cores_f" (num .CPUCores 1)}}</span></td>
<td class="num">{{if .MemoryBytes}}{{bytes .MemoryBytes}}{{else}}—{{end}}</td>
</tr>{{end}}</table>
{{else}}<p class="empty">No engines are placed on this node.</p>{{end}}
<p class="note">Engines: {{if .Node.Engines}}{{join .Node.Engines}}{{else}}none installed{{end}}{{if .Node.Models}} · {{len .Node.Models}} model file(s) on disk{{end}}.</p>
{{else}}<p class="empty">{{t "node.no_engines"}}</p>{{end}}
<p class="note">{{if .Node.Engines}}{{t "node.engines" (join .Node.Engines)}}{{else}}{{t "node.engines_none"}}{{end}}{{if .Node.Models}} · {{t "node.model_files" (len .Node.Models)}}{{end}}.</p>
</div>
{{if .HasHistory}}{{with .Charts}}
<h2>The last {{.Window}}</h2>
{{if .Err}}<p class="note error-row">No history: {{.Err}}</p>{{else}}
<h2>{{t "node.last" .Window}}</h2>
{{if .Err}}<p class="note error-row">{{t "dash.no_history" .Err}}</p>{{else}}
<div class="chart-row">
{{template "tschart" dict "t" .CPU "title" "CPU" "note" "All cores together, from the node's heartbeat."}}
{{template "tschart" dict "t" .Memory "title" "RAM in use" "note" "Total less what the kernel could hand to a new process."}}
{{template "tschart" dict "t" .CPU "title" "CPU" "note" (t "node.chart.cpu_note")}}
{{template "tschart" dict "t" .Memory "title" (t "node.chart.ram") "note" (t "node.chart.ram_note")}}
</div>
<div class="chart-row">
{{template "tschart" dict "t" .EngineCPU "title" "Cores busy, by model" "note" "Each engine's process group, the way top counts it."}}
{{template "tschart" dict "t" .EngineMemory "title" "Engine memory, by model" "note" "Resident memory, including the weights mapped from disk. The kernel can take those pages back, so this can exceed RAM in use."}}
{{template "tschart" dict "t" .EngineCPU "title" (t "node.chart.cores") "note" (t "node.chart.cores_note")}}
{{template "tschart" dict "t" .EngineMemory "title" (t "node.chart.engine_memory") "note" (t "node.chart.engine_memory_note")}}
</div>
{{if $.Node.GPUTotal}}
<div class="chart-row">
{{template "tschart" dict "t" .GPUUtil "title" "GPU busy, by card" "note" "Of each sample period, the share a kernel was running. A card held by a replica and idle here is one nobody is asking."}}
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory used, by card" "note" "Framebuffer in use: the weights and the KV cache an engine reserved, and anything not llmgw's."}}
{{template "tschart" dict "t" .GPUUtil "title" (t "node.chart.gpu_busy") "note" (t "node.chart.gpu_busy_note")}}
{{template "tschart" dict "t" .GPUMemory "title" (t "node.chart.gpu_memory") "note" (t "node.chart.gpu_memory_note")}}
</div>
<div class="chart-row">
{{template "tschart" dict "t" .GPUTemp "title" "GPU temperature, °C" "note" "Throttling usually starts in the high 80s; see what holds a card back in the table above."}}
{{template "tschart" dict "t" .GPUPower "title" "GPU power, W" "note" "Draw against the card's enforced limit; at the limit is sw_power_cap."}}
{{template "tschart" dict "t" .GPUTemp "title" (t "node.chart.gpu_temp") "note" (t "node.chart.gpu_temp_note")}}
{{template "tschart" dict "t" .GPUPower "title" (t "node.chart.gpu_power") "note" (t "node.chart.gpu_power_note")}}
</div>
{{end}}
<div class="chart-row">
{{template "tschart" dict "t" .Load "title" "Load average" "note" "Above the core count means work is waiting for a CPU."}}
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica" "note" "Requests accepted and not yet started, on the replicas here now."}}
{{template "tschart" dict "t" .Load "title" (t "node.chart.load") "note" (t "node.chart.load_note")}}
{{template "tschart" dict "t" .Queue "title" (t "dash.fleet.queue") "note" (t "node.chart.queue_note")}}
</div>
{{end}}
{{end}}{{else}}
<p class="note">No metrics store is configured, so there is no history here; <code>monitoring.prometheus</code> in the server configuration adds it.</p>
<p class="note">{{th "node.no_store"}}</p>
{{end}}
<footer><span>llmgw {{.Version}} · refreshes every {{.RefreshSec}}s</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}} · {{t "common.refreshes_every" .RefreshSec}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}
+9 -8
View File
@@ -1,13 +1,13 @@
{{define "node_logs"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
{{if .Follow}}<meta http-equiv="refresh" content="5">{{end}}
<title>llmgw — {{.Node}} logs</title><link rel="stylesheet" href="{{stylesheet}}">
<title>llmgw — {{t "logs.title" .Node}}</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>{{.Node}}</h1>
<p class="sub">{{if eq .Source "agent"}}the agent's log{{else}}output of engine <code>{{.Source}}</code>{{end}} · last {{.Count}} lines</p>
{{if .OnAdminListener}}<nav><a href="{{.Back}}">cluster</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
<p class="sub">{{if eq .Source "agent"}}{{t "logs.agent"}}{{else}}{{th "logs.engine" .Source}}{{end}} · {{t "logs.last_lines" .Count}}</p>
{{if .OnAdminListener}}<nav><a href="{{.Back}}">{{t "node.nav.cluster"}}</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
</header>
<p>
@@ -15,17 +15,18 @@
{{range .Sources}}<a href="{{$d.Base}}?source={{.}}&lines={{$d.Count}}{{if $d.Follow}}&follow=1{{end}}"{{if eq . $d.Source}} aria-current="page"><strong>{{.}}</strong>{{else}}>{{.}}{{end}}</a> &nbsp; {{end}}
</p>
<p class="note">
{{range .Counts}}<a href="{{$d.Base}}?source={{$d.Source}}&lines={{.}}{{if $d.Follow}}&follow=1{{end}}">{{.}}</a> {{end}}lines ·
{{if .Follow}}<a href="{{.Base}}?source={{.Source}}&lines={{.Count}}">stop following</a>{{else}}<a href="{{.Base}}?source={{.Source}}&lines={{.Count}}&follow=1">follow</a> (refresh every 5s){{end}}
{{range .Counts}}<a href="{{$d.Base}}?source={{$d.Source}}&lines={{.}}{{if $d.Follow}}&follow=1{{end}}">{{.}}</a> {{end}}{{t "logs.lines"}} ·
{{if .Follow}}<a href="{{.Base}}?source={{.Source}}&lines={{.Count}}">{{t "logs.stop"}}</a>{{else}}<a href="{{.Base}}?source={{.Source}}&lines={{.Count}}&follow=1">{{t "logs.follow"}}</a> {{t "logs.follow_every"}}{{end}}
</p>
{{/* An error from the control plane or the node is shown as it came. */}}
{{if .Error}}<p class="bad">{{.Error}}</p>{{end}}
{{if .Lines}}<pre id="log">{{range .Lines}}{{.}}
{{end}}</pre>
{{/* Newest at the bottom, like a terminal, and scrolled there: the line
worth reading is almost always the last one. */}}
<script>window.scrollTo(0, document.body.scrollHeight);</script>
{{else if not .Error}}<p class="empty">Nothing logged yet.</p>{{end}}
{{else if not .Error}}<p class="empty">{{t "logs.empty"}}</p>{{end}}
<footer class="note">llmgw {{.Version}} · {{.Now}}</footer>
<footer class="note">llmgw {{.Version}} · {{.Now}}{{template "langs"}}</footer>
</div></body></html>{{end}}
+18 -19
View File
@@ -1,54 +1,53 @@
{{define "nvlink"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<meta http-equiv="refresh" content="30">
<title>NVLink domains · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "nvlink.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap wide">
<header>
<h1>NVLink domains</h1>
<p class="sub">{{len .Domains}} multi-node domain{{if ne (len .Domains) 1}}s{{end}} — racks whose GPUs reach each other over NVLink across nodes</p>
{{if .OnAdminListener}}<nav><a href="/">cluster</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
<h1>{{t "nvlink.title"}}</h1>
<p class="sub">{{tn "nvlink.sub" (len .Domains)}}</p>
{{if .OnAdminListener}}<nav><a href="/">{{t "node.nav.cluster"}}</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
</header>
{{if not .Domains}}
<p class="empty">No GPU reports a multi-node NVLink domain. An eight-GPU machine's NVSwitch ends at its
chassis and is shown on its node's page; a domain appears here once the fabric manager has joined the
GPUs of an NVL72 rack, or an NVLink Switch system, to one.</p>
<p class="empty">{{t "nvlink.empty"}}</p>
{{end}}
<div class="racks">
{{range .Domains}}
<section class="rack{{if .Problems}} troubled{{end}}">
<header class="rack-head">
<div><h2>{{.Short}}</h2><p class="note"><code>{{.ClusterUUID}}</code> · clique {{.CliqueID}}</p></div>
<div><h2>{{.Short}}</h2><p class="note"><code>{{.ClusterUUID}}</code> · {{t "nvlink.clique" .CliqueID}}</p></div>
<div class="rack-stats">
<div><b>{{.Usable}}/{{len .Members}}</b><span>trays usable</span></div>
<div><b>{{.Joined}}/{{.GPUs}}</b><span>GPUs joined</span></div>
<div><b>{{.Usable}}/{{len .Members}}</b><span>{{t "nvlink.trays_usable"}}</span></div>
<div><b>{{.Joined}}/{{.GPUs}}</b><span>{{t "nvlink.gpus_joined"}}</span></div>
</div>
</header>
<div class="rack-body">
<div class="nvswitch"><span>NVLink Switch</span><i></i></div>
<div class="nvswitch"><span translate="no">NVLink Switch</span><i></i></div>
<div class="rack-trays">
<div class="backplane" aria-hidden="true"></div>
{{range .Members}}
<div class="tray{{if not .Usable}} out{{end}}">
<a class="tray-name" href="{{$.Nodes}}/{{.Node}}">{{.Node}}</a>
{{/* A GPU's fabric state and bandwidth are the driver's words. */}}
<div class="chips">{{range .GPUs}}<span class="gpu-chip{{if .Joined}} joined{{else}} lost{{end}}{{if and .Bandwidth (ne .Bandwidth "Full")}} slow{{end}}"
title="GPU {{.Index}} · {{.Name}} · {{or .State "no fabric"}}{{if .Bandwidth}} · {{.Bandwidth}} bandwidth{{end}}">{{.Index}}</span>{{end}}</div>
<span class="imex{{if and (eq .IMEX "active") .IMEXChannels}} up{{end}}" title="nvidia-imex {{or .IMEX "not installed"}}, {{.IMEXChannels}} channel(s)">IMEX</span>
title="GPU {{.Index}} · {{.Name}} · {{or .State (t "nvlink.no_fabric")}}{{if .Bandwidth}} · {{t "node.gpu.bandwidth" .Bandwidth}}{{end}}">{{.Index}}</span>{{end}}</div>
<span class="imex{{if and (eq .IMEX "active") .IMEXChannels}} up{{end}}" title="{{t "nvlink.imex" (or .IMEX (t "node.imex.not_installed")) .IMEXChannels}}" translate="no">IMEX</span>
</div>
{{end}}
</div>
</div>
{{/* The level is a class and a word; the message is the NVLink checker's
own, in English. */}}
{{if .Problems}}<ul class="fabric-problems rack-problems">{{range .Problems}}
<li class="{{.Level}}"><span class="tag{{if eq (printf "%s" .Level) "fail"}} bad{{end}}">{{.Level}}</span> {{.Message}}</li>{{end}}</ul>{{end}}
<li class="{{.Level}}"><span class="tag{{if eq (printf "%s" .Level) "fail"}} bad{{end}}">{{t (print "fabric.level." .Level)}}</span> {{.Message}}</li>{{end}}</ul>{{end}}
</section>
{{end}}
</div>
<p class="note">A replica that spans nodes is placed inside one of these first: tensor parallelism between its
nodes then runs over NVLink rather than InfiniBand. Only trays with every GPU joined and IMEX up are used —
a tray short of either would leave the replica a card, or its memory, short.</p>
<p class="note">{{t "nvlink.note"}}</p>
<footer><span>llmgw {{.Version}} · refreshes every 30s</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}} · {{t "common.refreshes_every" 30}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}
+40 -45
View File
@@ -1,112 +1,107 @@
{{define "org"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>{{.Org.Name}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>{{.Org.Name}}</h1>
<p class="sub">{{.People}} {{if eq .People 1}}person{{else}}people{{end}} ·
use, members and limits across the organization</p>
<p class="sub">{{tn "org.sub" .People}}</p>
{{template "nav" .Nav}}
</header>
{{if gt (len .Orgs) 1}}
<div class="filters">
<span class="filter-label">Organization</span>
<span class="filter-label">{{t "org.filter"}}</span>
{{range .Orgs}}<a class="chip{{if eq .ID $.Org.ID}} on{{end}}" href="/org/{{.ID}}">{{.Name}}</a>{{end}}
</div>
{{end}}
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
<h2>Use</h2>
<h2>{{t "org.use"}}</h2>
<div class="filters">
<span class="filter-label">Group by</span>
<span class="filter-label">{{t "usage.group_by"}}</span>
{{range .Groups}}<a class="chip{{if .Selected}} on{{end}}"
href="/org/{{$.Org.ID}}?by={{.Key}}&amp;window={{$.Window}}">{{.Label}}</a>{{end}}
<span class="filter-label">Window</span>
<span class="filter-label">{{t "usage.window"}}</span>
{{range .Windows}}<a class="chip{{if .Selected}} on{{end}}"
href="/org/{{$.Org.ID}}?by={{$.By}}&amp;window={{.Key}}">{{.Label}}</a>{{end}}
</div>
<div class="chart-row">
{{template "tschart" dict "t" .Requests "title" "Requests"
"note" "Requests made in this organization's projects."}}
{{template "tschart" dict "t" .Tokens "title" "Tokens"
"note" "Prompt and completion together."}}
{{template "tschart" dict "t" .Requests "title" (t "usage.requests")
"note" (t "org.requests_note")}}
{{template "tschart" dict "t" .Tokens "title" (t "usage.tokens")
"note" (t "org.tokens_note")}}
</div>
{{if .Priced}}
<div class="chart-row">
{{template "tschart" dict "t" .Spend "title" "Spend" "note" "In whole units, the ones a budget is written in."}}
{{template "tschart" dict "t" .Spend "title" (t "usage.spend") "note" (t "org.spend_note")}}
</div>
{{end}}
{{if .Rows}}
<table class="rows compact">
<tr><th>{{if eq .By "user"}}Person{{else if eq .By "project"}}Project{{else}}Team{{end}}</th>
<th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th></tr>
<tr><th>{{if eq .By "user"}}{{t "org.col.person"}}{{else if eq .By "project"}}{{t "org.col.project"}}{{else}}{{t "org.col.team"}}{{end}}</th>
<th class="num">{{t "usage.requests"}}</th><th class="num">{{t "usage.tokens"}}</th><th class="num">{{t "usage.spend"}}</th><th class="num">{{t "usage.errors"}}</th></tr>
{{range .Rows}}<tr>
<td>{{.Name}}</td><td class="num">{{compact .Requests}}</td><td class="num">{{compact .Tokens}}</td>
<td class="num">{{.Spend}}</td><td class="num">{{if .Errors}}{{.Errors}}{{else}}—{{end}}</td>
<td class="num">{{.Spend}}</td><td class="num">{{if .Errors}}{{count .Errors}}{{else}}—{{end}}</td>
</tr>{{end}}
</table>
{{else}}<p class="empty">Nothing used in this window.</p>{{end}}
<p class="note">Only what was spent in this organization's projects. What its
people do elsewhere — in another organization, or on their own — is not shown.</p>
{{else}}<p class="empty">{{t "org.use_empty"}}</p>{{end}}
<p class="note">{{t "org.use_scope"}}</p>
<h2>Teams, projects and limits</h2>
<h2>{{t "org.limits.title"}}</h2>
{{if .Quotas}}
<table class="rows quota-tree">
<tr><th>Where</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
<th class="num">Budget</th><th>Spent · 30 days</th></tr>
<tr><th>{{t "quotas.col.where"}}</th><th class="num">{{t "quotas.col.rpm"}}</th><th class="num">{{t "quotas.col.tpm"}}</th>
<th class="num">{{t "quotas.col.budget"}}</th><th>{{t "org.col.spent_30d"}}</th></tr>
{{range .Quotas}}
<tr class="q-depth-{{.Indent}}">
<td class="q-where"><span class="q-name">{{.Path}}</span> <span class="q-kind">{{.Scope}}</span>
{{if .Models}}<div class="q-sub">only <code>{{.Models}}</code></div>{{end}}
<td class="q-where"><span class="q-name">{{.Path}}</span> <span class="q-kind">{{t (print "quotas.kind." .Scope)}}</span>
{{if .Models}}<div class="q-sub">{{th "quotas.only_models" .Models}}</div>{{end}}
{{if .Shadowed}}<div class="q-sub q-warn">{{.Shadowed}}</div>{{end}}</td>
<td class="num">{{template "quota-value" dict "Own" .RPM "Up" .RPMUp "From" .RPMFrom}}</td>
<td class="num">{{template "quota-value" dict "Own" .TPM "Up" .TPMUp "From" .TPMFrom}}</td>
<td class="num">{{if .Budget}}{{.Budget}}{{if .Period}}<span class="q-per">/{{.Period}}</span>{{end}}
{{else}}{{template "quota-value" dict "Own" "" "Up" .BudgetUp "From" .BudgetFrom}}{{end}}</td>
<td class="num">{{template "quota-value" dict "Own" .Shown.RPM "Up" .RPMUp "From" .RPMFrom}}</td>
<td class="num">{{template "quota-value" dict "Own" .Shown.TPM "Up" .TPMUp "From" .TPMFrom}}</td>
<td class="num">{{if .Budget}}{{.Shown.Budget}}{{if .Period}}<span class="q-per">/{{.Shown.Period}}</span>{{end}}
{{else}}{{template "quota-value" dict "Up" .BudgetUp "From" .BudgetFrom}}{{end}}</td>
<td class="q-spent"><span class="q-spend">{{.Spend}}</span>
{{if .HasPct}}<span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span>
<span class="q-pct">{{.Percent}}%</span>{{end}}</td>
<span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}</td>
</tr>
{{end}}
</table>
<p class="note">Limits are set by the cluster's administrators.</p>
<p class="note">{{t "org.limits.note"}}</p>
{{end}}
<h2>Members</h2>
<h2>{{t "org.members.title"}}</h2>
{{if .Members}}
<table class="rows compact">
<tr><th>Person</th><th>Member of</th><th>State</th></tr>
<tr><th>{{t "org.col.person"}}</th><th>{{t "org.col.member_of"}}</th><th>{{t "org.col.state"}}</th></tr>
{{range .Members}}<tr>
<td>{{.Email}}</td>
<td>{{range .Places}}<span class="chip-m">{{.Where}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}</span>{{end}}</td>
<td class="state state-{{.State}}">{{.State}}</td>
<td>{{range .Places}}<span class="chip-m">{{.Where}}{{if eq .Role "manager"}} <b class="chip-role">{{t "org.role.manager"}}</b>{{end}}</span>{{end}}</td>
<td class="state state-{{.State}}">{{with .State}}{{t (print "org.state." .)}}{{end}}</td>
</tr>{{end}}
</table>
{{else}}<p class="empty">Nobody is a member yet.</p>{{end}}
{{else}}<p class="empty">{{t "org.members.empty"}}</p>{{end}}
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}
{{define "org-none"}}<!doctype html>
<html lang="en"><head>
<html lang="{{lang}}"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Organizations · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<title>{{t "org.none.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap">
<header>
<h1>Organizations</h1>
<p class="sub">There are none on this cluster</p>
<h1>{{t "org.none.title"}}</h1>
<p class="sub">{{t "org.none.sub"}}</p>
{{template "nav" .Nav}}
</header>
<p>Everyone uses the gateway on their own, under their personal quota. An
organization, once there is one, gets a page here for its managers: its use by
team, project and person, its limits, and its members.</p>
<p><a href="/admin/users">Create one on the people page</a>, then make someone
its manager with the star on their membership.</p>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span></footer>
<p>{{t "org.none.about"}}</p>
<p>{{th "org.none.how"}}</p>
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
</div></body></html>{{end}}