The manual in the binary: llmgw docs, and Help in the console
Every document and a page for every command, built from their help, read like info: topics, contents, find. ? opens the console's own page, which is new and covers every screen. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
44425193bd
commit
dac7ff0e99
7 files changed
+1129
-3
No files matched your search
+200
@@ -0,0 +1,200 @@
|
||||
# The console
|
||||
|
||||
`sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
|
||||
with nothing installed it opens the setup screens; on a control plane, the
|
||||
console described here; on a node, the node's own console, which edits
|
||||
`agent.yaml`. Everything on these screens can also be done by command — each
|
||||
screen says which — so a script never needs them and a person never needs
|
||||
to remember the commands.
|
||||
|
||||
The keyboard drives it; there is no mouse. **?** opens this page from
|
||||
anywhere, and the **Manual** entry holds every other page, as `llmgw docs`
|
||||
does without the console.
|
||||
|
||||
## The layout
|
||||
|
||||
A bar across the top names the machine. Down the left, the entries, in four
|
||||
groups: **Watch**, **People**, **Help**, **Settings** and **Do**. The panel on
|
||||
the right shows the entry chosen. The bottom line always says which keys do
|
||||
what at that moment, and a question or a result opens as a dialog over the
|
||||
panel.
|
||||
|
||||
| Key | Does |
|
||||
|---|---|
|
||||
| ↑ ↓ | Choose an entry, or a row in the panel |
|
||||
| → or Enter | Go into the panel |
|
||||
| Enter on a row | What can be done to it: its details first, then the rest |
|
||||
| ← or Esc | Back out: from a page to its list, from the panel to the entries |
|
||||
| A | Add one, on a view that holds things: a quota, an account, a key |
|
||||
| PgUp PgDn | Scroll a long panel |
|
||||
| ? | This page |
|
||||
| Q | Leave; the cluster keeps running |
|
||||
| Ctrl-C | Leave at once |
|
||||
|
||||
What is watched is read again every three seconds.
|
||||
|
||||
## Forms and questions
|
||||
|
||||
A change that needs values opens a form, filled in with what is set now. Tab
|
||||
or ↓ moves to the next field, ← → changes a choice, and Enter on **Apply**
|
||||
sends it; a form of one field is sent by Enter on it. A mistake the form can
|
||||
see — a maximum below the minimum, a word where a number goes — is said in the
|
||||
form and nothing is sent. One the control plane refuses comes back as a dialog
|
||||
with its reason, word for word.
|
||||
|
||||
Anything that cannot be undone — removing, draining, deleting, revoking —
|
||||
asks first, and only **Y** goes ahead.
|
||||
|
||||
Tasks that take a while and print as they go — serving a model, upgrading
|
||||
the cluster, a backup, adding a node — close the screen and run in the
|
||||
terminal, so their output is there to read; Enter brings the console back.
|
||||
|
||||
## Watch
|
||||
|
||||
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
|
||||
models, the store's members, and anything parked, unplaced or idle.
|
||||
|
||||
**Nodes** lists every node with its state, load, GPUs, replicas, disk and
|
||||
agent version. On a row:
|
||||
|
||||
- **Details** — its address, system, load, engines and GPUs, and any warning.
|
||||
- **Cordon** places nothing new on it; **Uncordon** undoes it.
|
||||
- **Drain** moves its replicas off one at a time, each replacement up before
|
||||
the original stops.
|
||||
- **vllm build / sglang build** (nodes with GPUs) — which build of the engine
|
||||
the node installs: `auto` follows its hardware, `none` installs none.
|
||||
- **Upgrade its agent** to a release; its engines keep serving.
|
||||
- **Delete model files nothing uses** frees the node's disk.
|
||||
- **Remove from the cluster** — the control plane forgets it; the machine is
|
||||
not touched.
|
||||
|
||||
**GPUs** is every card: how busy, its memory, temperature, power, and the
|
||||
model holding it.
|
||||
|
||||
**Models** lists the catalog: ready against wanted, status, class, profiles,
|
||||
and **UPDATE** when the registry has new weights for it. On a row:
|
||||
|
||||
- **Details** — description, replicas, class, owner, profiles, source, where
|
||||
its weights came from, what it reserves against what it uses, and its
|
||||
replicas.
|
||||
- **Replicas (scale)** — the minimum kept running, the most it scales to,
|
||||
and for a model loaded on demand how long it may sit unused. A maximum of
|
||||
0 stops it.
|
||||
- **Class** — critical, standard or opportunistic: whose capacity gives way
|
||||
when there is not enough for everyone.
|
||||
- **Owner** — the organization, team or project whose capacity quota it
|
||||
counts against.
|
||||
- **Resources** — the memory and cores one replica reserves on its node.
|
||||
- **LoRA adapters** — `name=source[@scale]`, separated by commas.
|
||||
- **Restart its replicas**, one by one or in place.
|
||||
- **Update to …** — when the repository it came from has published different
|
||||
weights for its files: they are fetched into the mirror beside the ones in
|
||||
use, checked, swapped in, and the replicas replaced one at a time.
|
||||
**Go back to …** returns to the weights the update replaced.
|
||||
**Ask the registry for new weights now** checks without waiting for the
|
||||
next scheduled look (`mirror.update_check`, six hours by default).
|
||||
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
|
||||
- **Try again**, for a model parked after failing every start.
|
||||
- **Remove** — its replicas stop and it leaves the catalog. A model from
|
||||
`server.yaml` is changed there instead; the console says so.
|
||||
|
||||
**A** on Models adds one from a definition file on this machine: the YAML a
|
||||
model takes in `server.yaml`.
|
||||
|
||||
**Replicas** lists every replica; on one, **Restart it**, replaced or in
|
||||
place.
|
||||
|
||||
**Usage** is the last 24 hours by team, project or organization (→ changes
|
||||
which): requests, tokens, errors and spend.
|
||||
|
||||
**Quotas** are capacity quotas: the GPUs, cores and memory each organization,
|
||||
team or project is guaranteed and may at most hold, and what it holds now.
|
||||
**A** adds one; on a row, change or remove it.
|
||||
|
||||
**Mirror** is what the model mirror holds, how far each download has got, and
|
||||
which models use each file. **A** downloads a URL into it; on a row, where
|
||||
nodes fetch it from, or remove it — with a warning when a model still uses it.
|
||||
|
||||
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
|
||||
coloured.
|
||||
|
||||
## People
|
||||
|
||||
The accounts, keys and organizations live in the store; these views read it
|
||||
and change it with the same commands as `llmgw users`, `llmgw keys` and
|
||||
`llmgw org`, so their checks and messages are the ones shown.
|
||||
|
||||
**Accounts** — who has signed up, their state, whether they see the cluster
|
||||
dashboard, and what they belong to. **A** creates an account, already
|
||||
approved. On a row: approve one waiting, suspend or reinstate, give or take
|
||||
the cluster dashboard, add to or take out of an organization, team or
|
||||
project, their limits, a new password (their sessions close), delete.
|
||||
|
||||
**API keys** — every key, its project, owner, state and last use. **A**
|
||||
issues one; the key is shown once, in the result, and never again. On a row:
|
||||
its limits, or revoke it.
|
||||
|
||||
**Organizations** — the tenancy tree: organizations, their teams, the teams'
|
||||
projects, with members, keys and limits. **A** adds an organization. On a
|
||||
row: add a team or project inside it, rename, move, limits for the level and
|
||||
for each of its members, its capacity quota, and for a project how long
|
||||
conversations and agent notes are kept. **Remove** takes away only something
|
||||
empty.
|
||||
|
||||
Limits are requests and tokens a minute, a budget per day or month, and the
|
||||
models allowed. Each level only narrows what is above it.
|
||||
|
||||
## Help
|
||||
|
||||
**Manual** lists every page there is: these, the installation and operation
|
||||
guides, the configuration reference, a page for every command, the README
|
||||
and the changes in each release. On a page:
|
||||
|
||||
| Key | Does |
|
||||
|---|---|
|
||||
| ↑ ↓, PgUp PgDn, Space | Scroll |
|
||||
| g, G | The top, the end |
|
||||
| T | Contents: the page's headings, Enter goes to one |
|
||||
| / | Find; the lines holding it are marked |
|
||||
| n, N | The next match, the one before |
|
||||
| [ ] | The topic before, the one after |
|
||||
| ← | Back to the list |
|
||||
|
||||
`llmgw docs` opens the same manual outside the console, on any machine,
|
||||
installed or not; `llmgw docs install` opens one page; piped, it prints the
|
||||
page's text.
|
||||
|
||||
## Settings
|
||||
|
||||
**server.yaml** (on a node, **agent.yaml**) is every key of the file, as a
|
||||
tree of menus built from the configuration reference: the value set, or the
|
||||
one that applies when nothing is, its type, and what it is for. Enter changes
|
||||
a value or opens a block; on a list or map of blocks, Enter opens an entry and
|
||||
**D** removes one. **U** puts a key back to its default. **S** saves: the file
|
||||
is checked the way the service reads it before anything is written, and its
|
||||
comments are kept. The service reads it when it starts, so the console offers
|
||||
to restart it. Another control plane keeps its own copy.
|
||||
|
||||
## Do
|
||||
|
||||
- **Add a node** — installs llmgw on a machine over ssh, checks it, gives it a
|
||||
certificate and starts its agent (`sudo llmgw node add`).
|
||||
- **Replicate the store** — moves the store onto two more machines under
|
||||
Patroni (offered only while it is not replicated).
|
||||
- **Serve a model** — Hugging Face's GGUF models: trending, most downloaded,
|
||||
most liked or newest (**S** changes which), or **/** to search. Enter on
|
||||
one lists its files with their sizes; the one chosen is fetched into the
|
||||
mirror, checked, added to the catalog and started, and a replica is asked
|
||||
to say hello (`llmgw ctl model serve`). `mirror.registry` points it at
|
||||
another registry, for a site without the internet.
|
||||
- **Upgrade the cluster** to a release: this machine, then every node in turn
|
||||
(`sudo llmgw ctl upgrade`).
|
||||
- **Restart all replicas**, one at a time.
|
||||
- **Back up now** — the store and `/etc/llmgw` into one archive
|
||||
(`sudo llmgw server backup`).
|
||||
- **Leave**.
|
||||
|
||||
## The node's console
|
||||
|
||||
On a node, `sudo llmgw` opens `agent.yaml` in the same settings view, and the
|
||||
manual. The cluster is watched and changed from a control plane.
|
||||
Reference in new issue
Block a user