llmgw/docs/console.md

The console

sudo llmgw, with nothing after it, on a machine llmgw runs on. On a machine with nothing installed it opens the setup screens; on a control plane, the console described here; on a node, the node's own console, which edits agent.yaml. Everything on these screens can also be done by command — each screen says which — so a script never needs them and a person never needs to remember the commands.

The keyboard drives it; there is no mouse. ? opens this page from anywhere, and the Manual entry holds every other page, as llmgw docs does without the console.

The layout

A bar across the top names the machine. Down the left, the entries, in four groups: Watch, People, Help, Settings and Do. The panel on the right shows the entry chosen. The bottom line always says which keys do what at that moment, and a question or a result opens as a dialog over the panel.

Key	Does
↑ ↓	Choose an entry, or a row in the panel
→ or Enter	Go into the panel
Enter on a row	What can be done to it: its details first, then the rest
← or Esc	Back out: from a page to its list, from the panel to the entries
A	Add one, on a view that holds things: a quota, an account, a key
PgUp PgDn	Scroll a long panel
?	This page
Q	Leave; the cluster keeps running
Ctrl-C	Leave at once

What is watched is read again every three seconds.

Forms and questions

A change that needs values opens a form, filled in with what is set now. Tab or ↓ moves to the next field, ← → changes a choice, and Enter on Apply sends it; a form of one field is sent by Enter on it. A mistake the form can see — a maximum below the minimum, a word where a number goes — is said in the form and nothing is sent. One the control plane refuses comes back as a dialog with its reason, word for word.

Anything that cannot be undone — removing, draining, deleting, revoking — asks first, and only Y goes ahead.

Serving a model and moving one to new weights are jobs on the control plane: started from here, they run there in the background whatever this screen does, and the console is free at once. Jobs follows them. Upgrading the cluster, a backup and adding a node change this machine or another over ssh; those close the screen and run in the terminal, so their output is there to read, and Enter brings the console back.

Watch

Overview is llmgw ctl status: nodes, GPUs by class, fabrics, replicas, models, the store's members, and anything parked, unplaced or idle.

Monitor draws the cluster the way btop draws a machine: CPU, memory, GPU busy and GPU memory, requests and tokens a second, time to first token, requests in flight and queued, network and disk — Braille charts, filled and coloured green to red by height — over 5 minutes, an hour, a day or a week (1–4, or R for the next). Below them each node's CPU, memory and network now and as a sparkline, each card's busy, memory, heat and power, and each model's requests a second. The history comes from the Prometheus named by monitoring.prometheus in server.yaml; without one the charts draw what this console has seen since it opened, and say so. Network and disk come from each machine's node exporter, which llmgw node add installs. N sends a test notification to each webhook in server.yaml's notify section and says how each answered (llmgw ctl notify test). A node's and a model's own pages have their charts too, over the same range.

Nodes lists every node with its state, load, GPUs, replicas, disk and agent version. On a row:

• Details — its address, system, load, engines and GPUs, and any warning.
• Cordon places nothing new on it; Uncordon undoes it.
• Drain moves its replicas off one at a time, each replacement up before the original stops.
• vllm build / sglang build (nodes with GPUs) — which build of the engine the node installs: auto follows its hardware, none installs none.
• Upgrade its agent to a release; its engines keep serving.
• Delete model files nothing uses frees the node's disk.
• Remove from the cluster — the control plane forgets it; the machine is not touched.

GPUs is every card: how busy, its memory, temperature, power, and the model holding it.

Models lists the catalog: ready against wanted, status, class, profiles, and UPDATE when the registry has new weights for it. On a row:

• Details — description, replicas, class, owner, profiles, source, where its weights came from, what it reserves against what it uses, and its replicas.
• Replicas (scale) — the minimum kept running, the most it scales to, and for a model loaded on demand how long it may sit unused. A maximum of 0 stops it.
• Class — critical, standard or opportunistic: whose capacity gives way when there is not enough for everyone.
• Owner — the organization, team or project whose capacity quota it counts against.
• Resources — the memory and cores one replica reserves on its node.
• LoRA adapters — name=source[@scale], separated by commas.
• Restart its replicas, one by one or in place.
• Update to … — when the repository it came from has published different weights for its files: they are fetched into the mirror beside the ones in use, checked, swapped in, and the replicas replaced one at a time. Go back to … returns to the weights the update replaced. Ask the registry for new weights now checks without waiting for the next scheduled look (mirror.update_check, six hours by default).
• Try the new weights as a canary… — instead of swapping them in, runs them beside the model as MODEL-canary, one replica, given the share of its requests you choose. With a canary running the menu has Canary's share…, Promote the canary… (the model moves to its weights, the canary is removed) and Take the canary out…. A canary that errs or answers more slowly than the model is taken out by itself. The MODEL column shows (10% to canary) and (canary of …).
• Why is it placed where it is — llmgw ctl explain-placement.
• Try again, for a model parked after failing every start.
• Remove — its replicas stop and it leaves the catalog. A model from server.yaml is changed there instead; the console says so.

A on Models adds one from a definition file on this machine: the YAML a model takes in server.yaml.

Replicas lists every replica; on one, Restart it, replaced or in place.

Usage is the last 24 hours by team, project or organization (→ changes which): requests, tokens, errors and spend.

Quotas are capacity quotas: the GPUs, cores and memory each organization, team or project is guaranteed and may at most hold, and what it holds now. A adds one; on a row, change or remove it.

Mirror is what the model mirror holds, one row for each version of each model — a file, or the set of files a model of several is: shards, a vision projector, a config and tokenizer — with how far its download has got and which models use it. A downloads a URL into it; on a row, its files with their checksums and where each was fetched from, the source a model definition names for it, or remove it — with a warning when a model still uses it.

Jobs lists the models being served or updated since the control plane started: what each is doing now — the download's progress, a replica starting — and whether it ended done, failed, or waiting for room on busy cards. The entry shows how many are running. Enter on one shows its steps. llmgw ctl jobs is the same, and llmgw ctl jobs ID follows one.

Logs shows a node's agent log, newest at the bottom, errors and warnings coloured.

People

The accounts, keys and organizations live in the store; these views read it and change it with the same commands as llmgw users, llmgw keys and llmgw org, so their checks and messages are the ones shown.

Accounts — who has signed up, their state, whether they see the cluster dashboard, and what they belong to. A creates an account, already approved. On a row: approve one waiting, suspend or reinstate, give or take the cluster dashboard, add to or take out of an organization, team or project, their limits, a new password (their sessions close), delete, and Forget them… for a request to be forgotten (llmgw users forget).

API keys — every key, its project, owner, state and last use. A issues one; the key is shown once, in the result, and never again. On a row: its limits, Rotate… — a new key with the same project, owner and limits, the old one working on for the overlap you give (llmgw keys rotate) — or revoke it.

Accounts rows also have Staff role…:

• viewer looks at every page under /admin and changes nothing;
• operator also runs the cluster from them (models through the mirror, a node's engines);
• billing sees usage and quotas;
• admin may do anything.

The STAFF column shows each person's role.

On Usage, E exports usage as CSV to a file on the control plane: by hour, day or month and any of organization, team, project, user, key and model.

Audit lists every change of the last week — through ctl, this console or the /admin pages, and people's own keys and passwords — with who made it, through what, and how it ended. Enter shows the fields it was given, with anything secret taken out. llmgw ctl audit is the same, with filters.

Organizations — the tenancy tree: organizations, their teams, the teams' projects, with members, keys and limits. A adds an organization. On a row: add a team or project inside it, rename, move, limits for the level and for each of its members, its capacity quota, and for a project how long conversations and agent notes are kept, and whether its requests are (Keep requests…: for how long, and what share), and whether they stay in the cluster (Requests stay in the cluster…: never an outside provider), and whether the same request is answered again from memory (Cache answers…: for how long, and whether only reproducible answers or any). Longest key lifetime… caps how long a key made there may live, and is the expiry of one made without any (llmgw keys ttl; auth.max_key_ttl in server.yaml caps every key). Remove takes away only something empty.

Limits are requests and tokens a minute, a budget per day or month, and the models allowed. Each level only narrows what is above it.

Guardrails… on a row lists the guardrail policies server.yaml defines. Each policy is marked in one of three ways:

• attached here (✓): Enter detaches it;
• applying from above (·): from everywhere, or from the organization or team named;
• neither: Enter attaches it.

The GUARDRAILS column names those attached to the row itself. Policies narrow like limits; see guardrails.md.

Help

Manual lists every page there is: these, the installation and operation guides, the configuration reference, a page for every command, the README and the changes in each release. On a page:

Key	Does
↑ ↓, PgUp PgDn, Space	Scroll
g, G	The top, the end
T	Contents: the page's headings, Enter goes to one
/	Find; the lines holding it are marked
n, N	The next match, the one before
[ ]	The topic before, the one after
←	Back to the list

llmgw docs opens the same manual outside the console, on any machine, installed or not; llmgw docs install opens one page; piped, it prints the page's text.

Settings

server.yaml (on a node, agent.yaml) is every key of the file, as a tree of menus built from the configuration reference: the value set, or the one that applies when nothing is, its type, and what it is for. Enter changes a value or opens a block; on a list or map of blocks, Enter opens an entry and D removes one. U puts a key back to its default. S saves: the file is checked the way the service reads it before anything is written, and its comments are kept. The service reads it when it starts, so the console offers to restart it. Another control plane keeps its own copy.

Do

• Add a node — installs llmgw on a machine over ssh, checks it, gives it a certificate and starts its agent (sudo llmgw node add).
• Replicate the store — moves the store onto two more machines under Patroni (offered only while it is not replicated).
• Serve a model — Hugging Face's models: trending, most downloaded, most liked or newest (S changes which), or / to search. F switches between GGUF files, which llama.cpp runs on the CPU or a GPU, and original safetensors weights, which vLLM runs on GPUs. Enter on a repository plans it against the cluster as it is, before anything is fetched: the model's shape is read from one file's header, or its config.json, and every file is listed with the best way it runs here — on one card, on several of a node's cards, across several nodes' cards over a fabric (original weights only), or on a CPU node — what each card or node would hold, and whether that hardware is free now. The file that suits the cluster best is first, marked ★; one that fits nowhere says so. Busy is not unfit: which file is best follows the hardware the cluster has, not what it is running, so a file for a card that is in use is still offered for it — served, it waits for the card, or with a fallback runs elsewhere until then. Choosing a file offers every way it fits, best first, and what to do when that one is busy: the next best way, written as a lower-priority profile so placement falls back to it, or wait. A vision model offers its projector too, fetched beside it and loaded with it so the model reads images. What is chosen is fetched into the mirror, checked, added to the catalog and started, and a replica is asked to say hello (llmgw ctl model serve; the plan alone is llmgw ctl model plan). mirror.registry points it at another registry, for a site without the internet.
• Upgrade the cluster to a release: this machine, then every node in turn (sudo llmgw ctl upgrade).
• Restart all replicas, one at a time.
• Back up now — the store and /etc/llmgw into one archive (sudo llmgw server backup).
• Leave.

The node's console

On a node, sudo llmgw opens agent.yaml in the same settings view, and the manual. The cluster is watched and changed from a control plane.