llmgw/docs/console.md

The console

sudo llmgw, with nothing after it, on a machine llmgw runs on. On a machine with nothing installed it opens the setup screens; on a control plane, the console described here; on a node, the node's own console, which shows what runs on that machine and edits agent.yaml. Everything on these screens can also be done by command — each screen says which — so a script never needs them and a person never needs to remember the commands.

The keyboard drives it; there is no mouse. ? opens this page from anywhere, and the Manual entry holds every other page, as llmgw docs does without the console.

The layout

A bar across the top names the machine. Down the left, the entries, in four groups: Watch, People, Help, Settings and Do. The panel on the right shows the entry chosen. The bottom line always says which keys do what at that moment, and a question or a result opens as a dialog over the panel.

Key	Does
↑ ↓	Choose an entry, or a row in the panel
→ or Enter	Go into the panel
Enter on a row	What can be done to it: its details first, then the rest
← or Esc	Back out: from a page to its list, from the panel to the entries
A	Add one, on a view that holds things: a quota, an account, a key
PgUp PgDn	Scroll a long panel
?	This page
Q	Leave; the cluster keeps running
Ctrl-C	Leave at once

What is watched is read again every three seconds.

Forms and questions

A change that needs values opens a form, filled in with what is set now. Tab or ↓ moves to the next field, ← → changes a choice, and Enter on Apply sends it; a form of one field is sent by Enter on it. A mistake the form can see — a maximum below the minimum, a word where a number goes — is said in the form and nothing is sent. One the control plane refuses comes back as a dialog with its reason, word for word.

Anything that cannot be undone — removing, draining, deleting, revoking — asks first, and only Y goes ahead.

Serving a model and moving one to new weights are jobs on the control plane: started from here, they run there in the background whatever this screen does, and the console is free at once. Jobs follows them. Upgrading the cluster, a backup and adding a node change this machine or another over ssh; those close the screen and run in the terminal, so their output is there to read, and Enter brings the console back.

Setting up

On a machine with nothing installed, the control plane is set up from one form (sudo llmgw server init): the administrator's email, other names nodes reach the machine by, and ticks for PostgreSQL, the model mirror, this machine as a node, the starter model and the guardrails' judge. Offline bundle and Its sha256 are for a site with no internet: the path of a bundle made by llmgw bundle create, and the sha256 it printed. Given, the bundle goes into the mirror, every file checked, and the cluster is set up to fetch nothing from outside (install.md, "Offline sites"). A bundle with no sha256, or with the mirror unticked, is said in the form and nothing starts.

Watch

Overview is llmgw ctl status: nodes, GPUs by class, fabrics, replicas, models, the store's members, and anything parked, unplaced or idle.

Monitor draws the cluster the way btop draws a machine: CPU, memory, GPU busy and GPU memory, requests and tokens a second, time to first token, requests in flight and queued, network and disk — Braille charts, filled and coloured green to red by height — over 5 minutes, an hour, a day or a week (1–4, or R for the next). Below them each node's CPU, memory and network now and as a sparkline, each card's busy, memory, heat and power, and each model's requests a second. The history comes from the Prometheus named by monitoring.prometheus in server.yaml; without one the charts draw what this console has seen since it opened, and say so. Network and disk come from each machine's node exporter, which llmgw node add installs. N sends a test notification to each webhook in server.yaml's notify section and says how each answered (llmgw ctl notify test). A node's and a model's own pages have their charts too, over the same range.

Nodes lists every node with its state, where it is (its rack, or row, room or site, and PDUs), load, GPUs, replicas, disk and agent version. A node in a held rack reads ready, held. On a row:

• Details — its address, location and any hold, system, load, engines and GPUs, and any warning.
• Cordon places nothing new on it; Uncordon undoes it.
• Location… sets its site, room, row, rack and PDUs (the llmgw.io/ labels), which placement spreads replicas across.
• Hold rack … places nothing new in the node's rack, and can drain it; Release … on a node in a held domain undoes it. See facility-aware-placement.md.
• Drain moves its replicas off one at a time, each replacement up before the original stops.
• vllm build / sglang build (nodes with GPUs) — which build of the engine the node installs: auto follows its hardware, none installs none.
• Upgrade its agent to a release; its engines keep serving.
• Delete model files nothing uses frees the node's disk.
• Remove from the cluster — the control plane forgets it; the machine is not touched.

GPUs is every card: how busy, its memory, temperature, power, and the model holding it.

Models lists the catalog: ready against wanted, status, class, profiles, and UPDATE when the registry has new weights for it. On a row:

• Details — description, replicas, class, owner, profiles, source, where its weights came from, what it reserves against what it uses, and its replicas.
• Replicas (scale) — the minimum kept running, the most it scales to, and for a model loaded on demand how long it may sit unused. A maximum of 0 stops it.
• Class — critical, standard or opportunistic: whose capacity gives way when there is not enough for everyone.
• Owner — the organization, team or project whose capacity quota it counts against.
• Resources — the memory and cores one replica reserves on its node.
• LoRA adapters — name=source[@scale], separated by commas.
• Restart its replicas, one by one or in place.
• Update to … — when the repository it came from has published different weights for its files: they are fetched into the mirror beside the ones in use, checked, swapped in, and the replicas replaced one at a time. Go back to … returns to the weights the update replaced. Ask the registry for new weights now checks without waiting for the next scheduled look (mirror.update_check, six hours by default).
• Try the new weights as a canary… — instead of swapping them in, runs them beside the model as MODEL-canary, one replica, given the share of its requests you choose. With a canary running the menu has Canary's share…, Promote the canary… (the model moves to its weights, the canary is removed) and Take the canary out…. A canary that errs or answers more slowly than the model is taken out by itself. The MODEL column shows (10% to canary) and (canary of …).
• Ask it a quality suite… — one of the suites the control plane has (in control.eval_dir, or built in: arithmetic, json-format), asked through the router; the dialog counts the answers as they come in, then shows the score, each case and the model's last runs. Esc stops watching; the run carries on and is kept. Its quality runs lists them, as llmgw ctl eval --history does.
• Hold its updates to a quality suite… — the suite and score its new weights must reach before an update or a promotion moves it to them (llmgw ctl model gate); none takes it away. With one set, Update to … and Promote the canary… say so.
• Why is it placed where it is — llmgw ctl explain-placement.
• Try again, for a model parked after failing every start.
• Remove — its replicas stop and it leaves the catalog. A model from server.yaml is changed there instead; the console says so.

A on Models adds one from a definition file on this machine: the YAML a model takes in server.yaml.

Replicas lists every replica; on one, Restart it, replaced or in place.

Usage is the last 24 hours by team, project or organization (→ changes which): requests, tokens, errors and spend.

Quotas are capacity quotas: the GPUs, cores and memory each organization, team or project is guaranteed and may at most hold, and what it holds now. A adds one; on a row, change or remove it.

Knowledge lists the knowledge-base collections: each one's owner, the model that embeds it, how many documents and chunks it holds and who else it is granted to. A makes one. On a row: its Documents, Grant… a project, team, organization, person or key read or write access, Revoke a grant, Answer from it by default… for a part of the tree, Try a search… as an administrator, or Delete… it. People fill collections from their own machines with llmgw-kb (knowledge.md). Off on a SQLite store, and says why.

Mirror is what the model mirror holds, one row for each version of each model — a file, or the set of files a model of several is: shards, a vision projector, a config and tokenizer — with how far its download has got and which models use it. A downloads a URL into it; I imports a model from an Ollama on this machine into a mirror directory on this machine — the mirror beside the control plane — and shows the model definition to add for it (a mirror elsewhere: llmgw mirror import-ollama there); on a row, its files with their checksums and where each was fetched from, the source a model definition names for it, or remove it — with a warning when a model still uses it.

Jobs lists the models being served or updated since the control plane started: what each is doing now — the download's progress, a replica starting — and whether it ended done, failed, or waiting for room on busy cards. The entry shows how many are running. Enter on one shows its steps. llmgw ctl jobs is the same, and llmgw ctl jobs ID follows one.

Logs shows a node's agent log, newest at the bottom, errors and warnings coloured.

People

The accounts, keys and organizations live in the store; these views read it and change it with the same commands as llmgw users, llmgw keys and llmgw org, so their checks and messages are the ones shown.

Accounts — who has signed up, their state, whether they see the cluster dashboard, and what they belong to. A creates an account, already approved. On a row: approve one waiting, suspend or reinstate, give or take the cluster dashboard, add to or take out of an organization, team or project, their limits, a new password (their sessions close), delete, and Forget them… for a request to be forgotten (llmgw users forget).

API keys — every key, its project, owner, state and last use. A issues one; the key is shown once, in the result, and never again. On a row: its limits, Rotate… — a new key with the same project, owner and limits, the old one working on for the overlap you give (llmgw keys rotate) — or revoke it.

Accounts rows also have Staff role…:

• viewer looks at every page under /admin and changes nothing;
• operator also runs the cluster from them (models through the mirror, a node's engines);
• billing sees usage and quotas;
• admin may do anything.

The STAFF column shows each person's role.

On Usage, E exports usage as CSV to a file on the control plane: by hour, day or month and any of organization, team, project, user, key and model.

Audit lists every change of the last week — through ctl, this console or the /admin pages, and people's own keys and passwords — with who made it, through what, and how it ended. Enter shows the fields it was given, with anything secret taken out. llmgw ctl audit is the same, with filters.

Organizations — the tenancy tree: organizations, their teams, the teams' projects, with members, keys and limits. A adds an organization. On a row: add a team or project inside it, rename, move, limits for the level and for each of its members, its capacity quota, and for a project how long conversations and agent notes are kept, and whether its requests are (Keep requests…: for how long, and what share), and whether they stay in the cluster (Requests stay in the cluster…: never an outside provider), and whether the same request is answered again from memory (Cache answers…: for how long, and whether only reproducible answers or any). Longest key lifetime… caps how long a key made there may live, and is the expiry of one made without any (llmgw keys ttl; auth.max_key_ttl in server.yaml caps every key). Remove takes away only something empty.

Limits are requests and tokens a minute, a budget per day or month, and the models allowed. Each level only narrows what is above it.

Sign-in — how people get in besides a password kept here: single sign-on, Active Directory and provisioning, each on or off, with the directory's last sync (how many people, accounts kept in step, suspended, reinstated).

• E sets up Entra ID. Give the tenant ID, the client ID and secret, and the groups for administrators and operators. A provider other than Entra takes its issuer URL in place of the tenant. The form shows the redirect address to register.
• D sets up Active Directory: the servers, where people are, the service account and its password, the domain's CA certificate as a PEM file on this machine, the group whose members may sign in, and groups for roles and teams.
• Before it is kept, the directory is tried from the control plane: each server reached and bound, how many people may sign in, how many are in each group. If any step fails, nothing is kept and the steps are shown. T tries the settings in force.
• On the provisioning row, Enter makes a SCIM token, shown once.
• The password, the client secret and the token are kept in the store, sealed by the master key when one is set. The settings name them as store:auth/…, so each control plane has them with no file put on it. The CA is written into the settings themselves.

Guardrails… on a row lists the guardrail policies server.yaml defines. Each policy is marked in one of three ways:

• attached here (✓): Enter detaches it;
• applying from above (·): from everywhere, or from the organization or team named;
• neither: Enter attaches it.

The GUARDRAILS column names those attached to the row itself. Policies narrow like limits; see guardrails.md.

Help

Manual lists every page there is: these, the installation and operation guides, the configuration reference, a page for every command, the README and the changes in each release. On a page:

Key	Does
↑ ↓, PgUp PgDn, Space	Scroll
g, G	The top, the end
T	Contents: the page's headings, Enter goes to one
/	Find; the lines holding it are marked
n, N	The next match, the one before
[ ]	The topic before, the one after
←	Back to the list

llmgw docs opens the same manual outside the console, on any machine, installed or not; llmgw docs install opens one page; piped, it prints the page's text.

Settings

server.yaml (on a node, agent.yaml) is every key, as a tree of menus built from the configuration reference: the value set, or the one that applies when nothing is, its type, and what it is for. Enter changes a value or opens a block; on a list or map of blocks, Enter opens an entry and D removes one. U puts a key back to its default. S saves.

On a control plane, the cluster's keys are read from and saved to the cluster's settings in the store — the same on every control plane, in force on all of them within seconds (llmgw ctl config does the same from a shell) — and this machine's own keys (listeners, certificates, the store's address) to its server.yaml, which the server notices and applies by itself. Either is checked before anything is kept, and the file's comments survive. On a node, agent.yaml is read when the agent starts, so the console offers to restart it.

Do

• Add a node — installs llmgw on a machine over ssh, checks it, gives it a certificate and starts its agent (sudo llmgw node add).
• Replicate the store — moves the store onto two more machines under Patroni (offered only while it is not replicated).
• Serve a model — Hugging Face's models: trending, most downloaded, most liked or newest (S changes which), or / to search. F switches between GGUF files, which llama.cpp runs on the CPU or a GPU, and original safetensors weights, which vLLM runs on GPUs. Enter on a repository plans it against the cluster as it is, before anything is fetched: the model's shape is read from one file's header, or its config.json, and every file is listed with the best way it runs here — on one card, on several of a node's cards, across several nodes' cards over a fabric (original weights only), or on a CPU node — what each card or node would hold, and whether that hardware is free now. The file that suits the cluster best is first, marked ★; one that fits nowhere says so. Busy is not unfit: which file is best follows the hardware the cluster has, not what it is running, so a file for a card that is in use is still offered for it — served, it waits for the card, or with a fallback runs elsewhere until then. Choosing a file offers every way it fits, best first, and what to do when that one is busy: the next best way, written as a lower-priority profile so placement falls back to it, or wait. A vision model offers its projector too, fetched beside it and loaded with it so the model reads images. What is chosen is fetched into the mirror, checked, added to the catalog and started, and a replica is asked to say hello (llmgw ctl model serve; the plan alone is llmgw ctl model plan). mirror.registry points it at another registry, for a site without the internet.
• Upgrade the cluster to a release: this machine, then every node in turn (sudo llmgw ctl upgrade).
• Restart all replicas, one at a time.
• Back up now — the store and /etc/llmgw into one archive (sudo llmgw server backup).
• Wipe this machine — removes llmgw's services, configuration, state and users from this machine, after listing them and asking for its name (sudo llmgw wipe; on a node too).
• Leave.

The node's console

On a node, sudo llmgw opens on Overview: what this machine is doing, read on the machine itself, so it works without an admin token and while the control plane is out of reach. Refreshed every three seconds, it shows

• llmgw-agent's state and since when, and the version of llmgw;
• each control plane the agent dials, from agent.yaml and the list the leader last sent, and whether it can be reached;
• each GPU: how busy, its memory, its heat, and the models on it;
• the engines running here, from the agent's own records: replica, model, engine, GPUs, port, whether the process is still there, and for how long;
• the free space where weights and engines are kept;
• the agent's warnings and errors of the last hour.

Below it, agent.yaml in the same settings view, and the manual. The cluster is watched and changed from a control plane.