llmgw/docs/console.md

# The console

`sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
with nothing installed it opens the setup screens; on a control plane, the
console described here; on a node, the node's own console, which shows
what runs on that machine and edits `agent.yaml`. Everything on these screens can also be done by command — each
screen says which — so a script never needs them and a person never needs
to remember the commands.

The keyboard drives it; there is no mouse. **?** opens this page from
anywhere, and the **Manual** entry holds every other page, as `llmgw docs`
does without the console.

## The layout

A bar across the top names the machine. Down the left, the entries, in four
groups: **Watch**, **People**, **Help**, **Settings** and **Do**. The panel on
the right shows the entry chosen. The bottom line always says which keys do
what at that moment, and a question or a result opens as a dialog over the
panel.

| Key | Does |
|---|---|
| ↑ ↓ | Choose an entry, or a row in the panel |
| → or Enter | Go into the panel |
| Enter on a row | What can be done to it: its details first, then the rest |
| ← or Esc | Back out: from a page to its list, from the panel to the entries |
| A | Add one, on a view that holds things: a quota, an account, a key |
| PgUp PgDn | Scroll a long panel |
| ? | This page |
| Q | Leave; the cluster keeps running |
| Ctrl-C | Leave at once |

What is watched is read again every three seconds.

## Forms and questions

A change that needs values opens a form, filled in with what is set now. Tab
or ↓ moves to the next field, ← → changes a choice, and Enter on **Apply**
sends it; a form of one field is sent by Enter on it. A mistake the form can
see — a maximum below the minimum, a word where a number goes — is said in the
form and nothing is sent. One the control plane refuses comes back as a dialog
with its reason, word for word.

Anything that cannot be undone — removing, draining, deleting, revoking —
asks first, and only **Y** goes ahead.

Serving a model and moving one to new weights are jobs on the control plane:
started from here, they run there in the background whatever this screen
does, and the console is free at once. **Jobs** follows them. Upgrading the
cluster, a backup and adding a node change this machine or another over ssh;
those close the screen and run in the terminal, so their output is there to
read, and Enter brings the console back.

## Setting up

On a machine with nothing installed, the control plane is set up from one
form (`sudo llmgw server init`): the administrator's email, other names nodes
reach the machine by, and ticks for PostgreSQL, the model mirror, this machine
as a node, the starter model and the guardrails' judge. **Offline bundle**
and **Its sha256** are for a site with no internet: the path of a bundle made
by `llmgw bundle create`, and the sha256 it printed. Given, the bundle goes
into the mirror, every file checked, and the cluster is set up to fetch
nothing from outside (install.md, "Offline sites"). A bundle with no sha256,
or with the mirror unticked, is said in the form and nothing starts.

## Watch

**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
models, the store's members, and anything parked, unplaced or idle.

**Monitor** draws the cluster the way btop draws a machine: CPU, memory,
GPU busy and GPU memory, requests and tokens a second, time to first token,
requests in flight and queued, network and disk — Braille charts, filled
and coloured green to red by height — over 5 minutes, an hour, a day or a
week (**1**–**4**, or **R** for the next). Below them each node's CPU,
memory and network now and as a sparkline, each card's busy, memory, heat
and power, and each model's requests a second. The history comes from the
Prometheus named by `monitoring.prometheus` in server.yaml; without one the
charts draw what this console has seen since it opened, and say so. Network
and disk come from each machine's node exporter, which `llmgw node add`
installs. **N** sends a test notification to each webhook in server.yaml's
`notify` section and says how each answered (`llmgw ctl notify test`). A node's and a model's own pages have their charts too, over the
same range.

**Nodes** lists every node with its state, where it is (its rack, or row,
room or site, and PDUs), load, GPUs, replicas, disk and agent version. A node
in a held rack reads *ready, held*. On a row:

- **Details** — its address, location and any hold, system, load, engines and
  GPUs, and any warning.
- **Cordon** places nothing new on it; **Uncordon** undoes it.
- **Location…** sets its site, room, row, rack and PDUs (the `llmgw.io/`
  labels), which placement spreads replicas across.
- **Hold rack …** places nothing new in the node's rack, and can drain it;
  **Release …** on a node in a held domain undoes it. See
  [facility-aware-placement.md](facility-aware-placement.md).
- **Drain** moves its replicas off one at a time, each replacement up before
  the original stops.
- **vllm build / sglang build** (nodes with GPUs) — which build of the engine
  the node installs: `auto` follows its hardware, `none` installs none.
- **Upgrade its agent** to a release; its engines keep serving.
- **Delete model files nothing uses** frees the node's disk.
- **Remove from the cluster** — the control plane forgets it; the machine is
  not touched.

**GPUs** is every card: how busy, its memory, temperature, power, and the
model holding it.

**Models** lists the catalog: ready against wanted, status, class, profiles,
and **UPDATE** when the registry has new weights for it. On a row:

- **Details** — description, replicas, class, owner, profiles, source, where
  its weights came from, what it reserves against what it uses, and its
  replicas.
- **Replicas (scale)** — the minimum kept running, the most it scales to,
  and for a model loaded on demand how long it may sit unused. A maximum of
  0 stops it.
- **Class** — critical, standard or opportunistic: whose capacity gives way
  when there is not enough for everyone.
- **Owner** — the organization, team or project whose capacity quota it
  counts against.
- **Resources** — the memory and cores one replica reserves on its node.
- **LoRA adapters** — `name=source[@scale]`, separated by commas.
- **Restart its replicas**, one by one or in place.
- **Update to …** — when the repository it came from has published different
  weights for its files: they are fetched into the mirror beside the ones in
  use, checked, swapped in, and the replicas replaced one at a time.
  **Go back to …** returns to the weights the update replaced.
  **Ask the registry for new weights now** checks without waiting for the
  next scheduled look (`mirror.update_check`, six hours by default).
- **Try the new weights as a canary…** — instead of swapping them in, runs
  them beside the model as `MODEL-canary`, one replica, given the share of
  its requests you choose. With a canary running the menu has **Canary's
  share…**, **Promote the canary…** (the model moves to its weights, the
  canary is removed) and **Take the canary out…**. A canary that errs or
  answers more slowly than the model is taken out by itself. The MODEL
  column shows `(10% to canary)` and `(canary of …)`.
- **Ask it a quality suite…** — one of the suites the control plane has (in
  `control.eval_dir`, or built in: `arithmetic`, `json-format`), asked
  through the router; the dialog counts the answers as they come in, then
  shows the score, each case and the model's last runs. Esc stops watching;
  the run carries on and is kept. **Its quality runs** lists them, as
  `llmgw ctl eval --history` does.
- **Hold its updates to a quality suite…** — the suite and score its new
  weights must reach before an update or a promotion moves it to them
  (`llmgw ctl model gate`); **none** takes it away. With one set, **Update
  to …** and **Promote the canary…** say so.
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
- **Try again**, for a model parked after failing every start.
- **Remove** — its replicas stop and it leaves the catalog. A model from
  `server.yaml` is changed there instead; the console says so.

**A** on Models adds one from a definition file on this machine: the YAML a
model takes in `server.yaml`.

**Replicas** lists every replica; on one, **Restart it**, replaced or in
place.

**Usage** is the last 24 hours by team, project or organization (→ changes
which): requests, tokens, errors and spend.

**Quotas** are capacity quotas: the GPUs, cores and memory each organization,
team or project is guaranteed and may at most hold, and what it holds now.
**A** adds one; on a row, change or remove it.

**Knowledge** lists the knowledge-base collections: each one's owner, the
model that embeds it, how many documents and chunks it holds and who else it
is granted to. **A** makes one. On a row: its **Documents**, **Grant…** a
project, team, organization, person or key read or write access, **Revoke**
a grant, **Answer from it by default…** for a part of the tree, **Try a
search…** as an administrator, or **Delete…** it. People
fill collections from their own machines with `llmgw-kb`
([knowledge.md](knowledge.md)). Off on a SQLite store, and says why.

**MCP servers** lists the company's own MCP servers that agents reach through
the gateway's `/mcp`, their tools named `server.tool`: whether each is on,
who has been granted its tools, its rate per caller, the names of the
credentials it is sent (never their values) and its address. **A** adds one:
its name, which is its tools' prefix, its address, and a bearer token or
another header, kept sealed in the store. On a row: **Test, and list its
tools**, with its credential, as a call would; **Grant its tools to…** a
team, project, organization, person or key — all of them, `jira.*`, or some
by name — beside whatever else that level is granted, and saying when a
level above keeps them out of reach; **Revoke** a grant;
**Change…** its address, timeout or rate; **Replace its credential…**;
**Turn off** (its tools are listed nowhere and every call refused) or on;
**Remove…**. A server's tools are seen nowhere until granted. The screen
also says how a person points an agent at the gateway, with a key of their
own from `/profile/keys`. `llmgw ctl mcp` does the same from a shell, and
the web page **MCP servers** (`/admin/mcp`, administrators only) in a browser.

**Mirror** is what the model mirror holds, one row for each version of each
model — a file, or the set of files a model of several is: shards, a vision
projector, a config and tokenizer — with how far its download has got and
which models use it. **A** downloads a URL into it; **I** imports a model
from an Ollama on this machine into a mirror directory on this machine — the
mirror beside the control plane — and shows the model definition to add for
it (a mirror elsewhere: `llmgw mirror import-ollama` there); on a row, its files with
their checksums and where each was fetched from, the source a model
definition names for it, or remove it — with a warning when a model still
uses it.

**Jobs** lists the models being served or updated since the control plane
started: what each is doing now — the download's progress, a replica
starting — and whether it ended done, failed, or waiting for room on busy
cards. The entry shows how many are running. Enter on one shows its steps.
`llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one.

**Logs** shows a node's agent log, newest at the bottom, errors and warnings
coloured.

## People

The accounts, keys and organizations live in the store; these views read it
and change it with the same commands as `llmgw users`, `llmgw keys` and
`llmgw org`, so their checks and messages are the ones shown.

**Accounts** — who has signed up, their state, whether they see the cluster
dashboard, and what they belong to. **A** creates an account, already
approved. On a row: approve one waiting, suspend or reinstate, give or take
the cluster dashboard, add to or take out of an organization, team or
project, their limits, a new password (their sessions close), delete, and
**Forget them…** for a request to be forgotten (`llmgw users forget`).

**API keys** — every key, its project, owner, state and last use. **A**
issues one; the key is shown once, in the result, and never again. On a row:
its limits, **Rotate…** — a new key with the same project, owner and limits,
the old one working on for the overlap you give (`llmgw keys rotate`) — or
revoke it.

**Accounts** rows also have **Staff role…**:
- `viewer` looks at every page under /admin and changes nothing;
- `operator` also runs the cluster from them (models through the mirror, a
  node's engines);
- `billing` sees usage and quotas;
- `admin` may do anything.

The STAFF column shows each person's role.

On **Usage**, **E** exports usage as CSV to a file on the control plane:
by hour, day or month and any of organization, team, project, user, key and
model.

**Audit** lists every change of the last week — through `ctl`, this console
or the /admin pages, and people's own keys and passwords — with who made
it, through what, and how it ended. Enter shows the fields it was given,
with anything secret taken out. `llmgw ctl audit` is the same, with filters.

**Organizations** — the tenancy tree: organizations, their teams, the teams'
projects, with members, keys and limits. **A** adds an organization. On a
row: add a team or project inside it, rename, move, limits for the level and
for each of its members, its capacity quota, and for a project how long
conversations and agent notes are kept, and whether its requests are
(**Keep requests…**: for how long, and what share), and whether they stay in
the cluster (**Requests stay in the cluster…**: never an outside provider),
and whether the same request is answered again from memory (**Cache
answers…**: for how long, and whether only reproducible answers or any). **Longest key lifetime…** caps how
long a key made there may live, and is the expiry of one made without any
(`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key).
**Remove** takes away only something empty.

Limits are requests and tokens a minute, a budget per day or month, and the
models allowed. Each level only narrows what is above it.

**Sign-in** — how people get in besides a password kept here: single
sign-on, Active Directory and provisioning, each on or off, with the
directory's last sync (how many people, accounts kept in step, suspended,
reinstated).
- **E** sets up Entra ID. Give the tenant ID, the client ID and secret,
  and the groups for administrators and operators. A provider other than
  Entra takes its issuer URL in place of the tenant. The form shows the
  redirect address to register.
- **D** sets up Active Directory: the servers, where people are, the
  service account and its password, the domain's CA certificate as a PEM
  file on this machine, the group whose members may sign in, and groups
  for roles and teams.
- Before it is kept, the directory is tried from the control plane: each
  server reached and bound, how many people may sign in, how many are in
  each group. If any step fails, nothing is kept and the steps are shown.
  **T** tries the settings in force.
- On the provisioning row, Enter makes a SCIM token, shown once.
- The password, the client secret and the token are kept in the store,
  sealed by the master key when one is set. The settings name them as
  `store:auth/…`, so each control plane has them with no file put on it.
  The CA is written into the settings themselves.

**Guardrails…** on a row lists the guardrail policies server.yaml defines.
Each policy is marked in one of three ways:

- attached here (✓): Enter detaches it;
- applying from above (·): from everywhere, or from the organization or
  team named;
- neither: Enter attaches it.

The **GUARDRAILS** column names those attached to the row itself. Policies
narrow like limits; see [guardrails.md](guardrails.md).

## Help

**Manual** lists every page there is: these, the installation and operation
guides, the configuration reference, a page for every command, the README
and the changes in each release. On a page:

| Key | Does |
|---|---|
| ↑ ↓, PgUp PgDn, Space | Scroll |
| g, G | The top, the end |
| T | Contents: the page's headings, Enter goes to one |
| / | Find; the lines holding it are marked |
| n, N | The next match, the one before |
| [ ] | The topic before, the one after |
| ← | Back to the list |

`llmgw docs` opens the same manual outside the console, on any machine,
installed or not; `llmgw docs install` opens one page; piped, it prints the
page's text.

## Settings

**server.yaml** (on a node, **agent.yaml**) is every key, as a tree of menus
built from the configuration reference: the value set, or the one that
applies when nothing is, its type, and what it is for. Enter changes a value
or opens a block; on a list or map of blocks, Enter opens an entry and **D**
removes one. **U** puts a key back to its default. **S** saves.

On a control plane, the cluster's keys are read from and saved to the
cluster's settings in the store — the same on every control plane, in force
on all of them within seconds (`llmgw ctl config` does the same from a
shell) — and this machine's own keys (listeners, certificates, the store's
address) to its server.yaml, which the server notices and applies by itself.
Either is checked before anything is kept, and the file's comments survive.
On a node, agent.yaml is read when the agent starts, so the console offers
to restart it.

## Do

- **Add a node** — installs llmgw on a machine over ssh, checks it, gives it a
  certificate and starts its agent (`sudo llmgw node add`).
- **Replicate the store** — moves the store onto two more machines under
  Patroni (offered only while it is not replicated).
- **Serve a model** — Hugging Face's models: trending, most downloaded,
  most liked or newest (**S** changes which), or **/** to search. **F**
  switches between GGUF files, which llama.cpp runs on the CPU or a GPU, and
  original safetensors weights, which vLLM runs on GPUs. Enter on a
  repository plans it against the cluster as it is, before anything is
  fetched: the model's shape is read from one file's header, or its
  config.json, and every file is listed with the best way it runs here — on
  one card, on several of a node's cards, across several nodes' cards over a
  fabric (original weights only), or on a CPU node — what each card or node
  would hold, and whether that hardware is free now. The file that suits the
  cluster best is first, marked ★; one that fits nowhere says so. Busy is not
  unfit: which file is best follows the hardware the cluster has, not what it
  is running, so a file for a card that is in use is still offered for it —
  served, it waits for the card, or with a fallback runs elsewhere until then. Choosing a
  file offers every way it fits, best first, and what to do when that one is
  busy: the next best way, written as a lower-priority profile so placement
  falls back to it, or wait. A vision model offers its projector too, fetched
  beside it and loaded with it so the model reads images. What is chosen is
  fetched into the mirror, checked, added to the catalog and started, and a
  replica is asked to say hello (`llmgw ctl model serve`; the plan alone is
  `llmgw ctl model plan`).
  `mirror.registry` points it at another registry, for a site without the
  internet.
- **Upgrade the cluster** to a release: this machine, then every node in turn
  (`sudo llmgw ctl upgrade`).
- **Restart all replicas**, one at a time.
- **Back up now** — the store and `/etc/llmgw` into one archive
  (`sudo llmgw server backup`).
- **Wipe this machine** — removes llmgw's services, configuration, state and
  users from this machine, after listing them and asking for its name
  (`sudo llmgw wipe`; on a node too).
- **Leave**.

## The node's console

On a node, `sudo llmgw` opens on **Overview**: what this machine is doing, read
on the machine itself, so it works without an admin token and while the control
plane is out of reach. Refreshed every three seconds, it shows

- llmgw-agent's state and since when, and the version of llmgw;
- each control plane the agent dials, from `agent.yaml` and the list the
  leader last sent, and whether it can be reached;
- each GPU: how busy, its memory, its heat, and the models on it;
- the engines running here, from the agent's own records: replica, model,
  engine, GPUs, port, whether the process is still there, and for how long;
- the free space where weights and engines are kept;
- the agent's warnings and errors of the last hour.

Below it, `agent.yaml` in the same settings view, and the manual. The cluster
is watched and changed from a control plane.