Files
llmgw/docs/onboarding.md
T
Ilker ManapandClaude Opus 5.5 098886e0e4 Let a person belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:

- memberships: a person is a member of any number of organizations,
  teams and projects, or of none; a member of a team or organization
  may use every project in it. users join/leave, and the people page
  adds and takes out from either side. Approval no longer needs a
  project.
- personal keys: a key may have an owner and no project. /profile asks
  what a new key spends against; keys create --for does the same.
  auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
  project and the chat page; --per-member on an org, team or project
  gives each member their own share there. Resolution folds the tree,
  each per-member limit for the key's owner, and the owner's own, each
  in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
  personal use and chat included; restore no longer drops rows with no
  project.

Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.

Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.

users move is gone; join and leave replace it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 19:03:46 +02:00

15 KiB

Onboarding a user

Written for whoever operates an llmgw deployment: the person with a shell on the control plane machine.

There are two ways a person gets a key. Pick one and put it in the config.

Self-service Operator-issued
Config auth.signup_open: true auth.signup_open: false
They do sign up, wait, then make their own keys wait for a link
You do approve once issue every key
Good for more than a handful of people a few, or machines

Self-service does not mean open. A signup produces an account that can do nothing at all; approval is what makes it work, and approval needs a shell here rather than a session cookie, because it is the step that decides who gets to spend the cluster.

Everything below runs as the user the server runs as, so the files it creates stay readable by it:

sudo -u llmgw llmgw <command> --store /var/lib/llmgw/llmgw.db

Self-service

Turn on the form:

auth:
  enabled: true
  store: /var/lib/llmgw/llmgw.db
  signup_open: true

The landing page then links to it, and the rest is:

llmgw users list --pending --store …
ID          EMAIL              STATE    ADMIN  MEMBER OF     SIGNED UP
usr-k4mq…   ayse@example.com   pending  -      on their own  4m12s ago

llmgw users approve ayse@example.com --by ilker --store …
llmgw users approve ayse@example.com --project Acme/ml/training --store …   # or straight into a project

That is the whole operator loop. From then on they sign in at /login, and /profile is where they create keys, see when each was last used, and revoke the ones they are done with. A key is shown once on the page that creates it and never again.

This gateway sends no email. There is no "your account was approved" message and no password reset, so tell people what to expect: they sign up, they tell you, you approve, they sign in. If that loop is too manual for your size, it is the thing to automate — not the approval step.

To close an account:

llmgw users suspend usr-k4mq… --store …

which closes their sessions immediately. Their API keys keep working — those are separate credentials, and revoking them is a separate step:

llmgw keys list --project default --store …
llmgw keys revoke key-… --store …

That separation is deliberate: a suspended person's running jobs should not all fail at once without someone deciding they should.

Operator-issued

With signup_open: false there is no form, and you issue keys yourself. This is also the path for things that are not people — CI, a notebook server, a script.

Decide where the key belongs

Keys hang off a project, projects off a team, teams off an organization. Limits can be set at any of those levels and compose downward by taking the most restrictive value — a key cannot raise what its team set, and a team cannot raise what its organization set. That is the only rule under which a budget means anything.

If you have no structure yet, one of each is fine:

llmgw org add org mynodes
llmgw org add team default --in mynodes
llmgw org add project default --in mynodes/default
llmgw org tree --people

Everything that takes an organization, team or project — here, in users and in keys — accepts its id, its path (mynodes/default/default) or its name when the name is unique, so an id never has to be copied off a page. When there is only one project, --project can be left out.

The tree is not fixed once made:

llmgw org rename project Acme/platform/gateway api-gateway
llmgw org move project Acme/platform/gateway --to Scania/ml
llmgw org remove team Acme/old                 # only when nothing is in it
llmgw users delete ayse@example.com --yes      # signs out, revokes their keys
llmgw keys list --all                          # every key, where it spends, and whose

A project that ever had a key cannot be removed, because usage history refers to it — rename it instead.

Where people belong

A person can be a member of any number of organizations, teams and projects, or of none:

llmgw users join ayse@example.com --team Acme/ml          # every project in ml
llmgw users join ayse@example.com --project Scania/research/vision
llmgw users leave ayse@example.com --project Acme/ml/training

A member of a team or organization may use every project in it. Leaving revokes the keys the person made in projects they can no longer reach; a project they still reach through another membership keeps them.

Someone who is a member of nothing still uses the gateway, on their own: personal keys, and the chat page, within their personal limits. When they make a key on /profile they choose what it spends against — one of their projects, or "Personal". An operator can do the same:

llmgw keys create laptop --for ayse@example.com                      # personal
llmgw keys create ci --for ayse@example.com --project Acme/ml/training

Personal use can be limited by default, or turned off so people use the gateway only through their projects:

auth:
  personal:
    rpm: 20
    tpm: 50000
    budget: 10        # whole units
    period: month
    # disabled: true

A personal key has no project, so it keeps no conversations or agent notes; those are kept per project.

llmgw keys init still works for provisioning scripts and is idempotent, but it will no longer move a team or project that already exists under another parent; it says so and stops.

Split it when you want separate limits or separate reporting, not before.

llmgw keys create --store … \
  --project default \
  --band gold \
  --base-url https://llmgw.example \
  --claim \
  "ayse-laptop"

https://llmgw.example/claim/3mKq…

Send the link, not the key.

A key pasted into a chat window stays in that chat window forever, searchable by everyone who joins the channel later. A link works once and then expires, so the worst case is someone opening it before the intended person — which you find out about, because they will tell you the link is dead.

The name is for you, not the user: it appears in keys list and is how you will know which key to revoke.

Why the link is safe to store. The key is sealed with the link's own token, and the token is never written down — only its hash is. A database leak yields two useless hashes and a blob nothing in the database can open.

Without --claim the key is printed to stdout instead, for when you are provisioning something that is not a person:

llmgw keys create --store … --project ci "github-runner" > /run/secrets/llmgw-key

Limits

Limits are optional. A key with none is unlimited, which is the right default on a private cluster and the wrong one the moment the gateway faces anything else.

# Everyone in this project
llmgw keys limits --store … --scope project --id default \
  --rpm 60 --tpm 200000

# One key, tighter
llmgw keys limits --store … --scope key --id key-mpmip77twpt7qaxc \
  --rpm 10

# Restrict which models
llmgw keys limits --store … --scope project --id default \
  --models qwen-0.5b,llama-3.3-70b

An absent model list permits everything. An empty one permits nothing — --models "" is a way to suspend a project without revoking its keys.

Allowlists compose by intersection: a key cannot add a model its team withheld. A model's canonical name and any of its aliases name the same thing, so either form works.

Budgets are given in whole units, as the quotas page shows them, and what a unit means is yours to decide. Self-hosted models have no per-token price, so it can be GPU-seconds, a notional rate, or one unit per thousand tokens everywhere — a price of 1 000 000 in auth.pricing, which is in micro-units — and a budget becomes a plain quota of thousands of tokens. Models are priced in the server's auth.pricing section; a model priced at zero is exempt from budget checks, which is what lets a key with nothing left still be served a cheap fallback.

llmgw keys limits --store … --scope project --id Acme/platform/gateway \
  --budget 50 --period month              # 50 units a month

Two more kinds of limit follow people rather than the tree:

# A person, everywhere: every key they hold, every project, the chat page
llmgw keys limits --scope user --id ayse@example.com --budget 25

# Each member of a project separately — a fair share, not one shared pool
llmgw keys limits --scope project --id Acme/ml/training --per-member --rpm 20

A request is held to all of them at once: the key's own, its project's, team's and organization's, each per-member limit on those for its owner, and the owner's own. The strictest wins, and each is counted in its own pool.

The landing page

Someone who reaches the gateway without a key sees the landing page. Tell it who to send them to, or it can only shrug:

landing:
  contact: "platform@example.com"   # or an https:// URL, or "#platform on Slack"
  notice: "Internal use only."

An address becomes a mailto link, an http(s) URL stays a link, anything else is shown as text. Other schemes are never linked — configuration is not always written by whoever runs the gateway, and a javascript: href on a page served to every visitor is a short path to a stored XSS.

What the user sees

They open the link, press the button, and get the key once. The page shows them the base URL and a working curl, and points at two endpoints:

  • GET /v1/models — what this key may use
  • GET /me — its limits and what it has spent

/me is there so nobody has to ask you "why was I rate limited" or "how much have I used". It is outside /v1 because it is not part of the OpenAI API.

Rotation and revocation

llmgw keys list --store … --project default

LAST USED is written in batches rather than per request, so it lags by up to a minute. It is the column that tells you whether a key is still in use, which is what makes rotation safe:

To rotate, issue the new key, let both work while the user switches, watch LAST USED on the old one stop moving, then revoke it.

llmgw keys revoke --store … key-mpmip77twpt7qaxc

Revocation is a soft delete: the key stops working, the row stays, and its usage history keeps its subject. A running server keeps accepting a revoked key until its cache expires — auth.cache_ttl, 30 seconds by default. That is the price of keeping the database off the request path, and it is the number to lower if you need revocation to bite faster.

Passwords

People change their own from /profile. It asks for the current one — a session is what someone gets from a borrowed laptop, and the password is what they do not have — and it signs every other browser out, which is the point if they are changing it because something went wrong.

There is no self-service reset, because this gateway sends no email and a reset link with nowhere to send it is not a reset. When someone loses theirs:

llmgw users reset-password usr-k4mq… --password "a temporary passphrase" --store …

Tell them out of band and have them change it from their profile. Their sessions are closed — if they lost the password, the reason might be that someone else has it. Their API keys keep working; those are separate credentials.

Trying a model in the browser

/chat is a playground for anyone signed in. It exists so somebody can see whether a model works without setting up a client, and — more often — so they can tell whether a problem is the gateway or their code.

The browser never holds a key. The page carries a session; the server turns that into the same identity an API key would produce, resolved from the person's project, and the request goes down the ordinary path: admission, model allowlist, rate limits, budget, accounting. Anything else would make the page a hole in every limit the rest of the gateway enforces.

Usage from the page is attributed to session:<user-id> rather than a key, so it is billed to the right project and visibly did not come from a credential somebody holds. One turn is capped at 512 tokens: a playground should not be the thing that ties up a replica for ten minutes, and anyone who wants a longer answer has an API key for that.

Nothing is stored. The conversation lives in the page and is gone when it is closed.

Seeing the cluster

The dashboard — nodes, replicas, models, why something did not get placed, usage per key — is served twice, and the difference matters.

On the admin listener (127.0.0.1:9090 by default) it needs no authentication: having a shell on the machine is the credential.

ssh -L 9090:127.0.0.1:9090 <control-plane>   # then http://127.0.0.1:9090/

That listener also serves pprof, which can dump the process heap — where API keys sit in plaintext. Never expose that port. It is bound to loopback for that reason.

On the client-facing port the same page is at /admin, behind a session and an admin flag, so it can be reached from a browser without the port carrying pprof along with it:

llmgw users promote usr-k4mq… --store …

Someone without the flag gets a 404 rather than a 403, because 403 confirms the page exists.

Managing accounts from the browser

/admin/users is the tree: organizations, teams and projects, each project opening onto its members and its keys. From there an admin approves people (into a project or on their own), adds them to and takes them out of organizations, teams and projects, suspends and removes them, revokes keys, and adds, renames, moves and removes organizations, teams and projects. /admin/quotas sets limits on the tree, per member of any part of it, and per person. A searchable list of every account is at the bottom. The page works as plain forms; with scripts on, a change updates it in place and the projects you had open stay open.

Three things are deliberately not there:

Why
llmgw users promote An admin who can make admins turns a stolen session into permanent access
llmgw users reset-password Setting someone's password is taking over their account
llmgw keys create Issuing a credential on another person's behalf

Each of those turns a stolen admin cookie into more than it was, and each is rare enough that a shell is no hardship. Cordon, drain and everything else that touches the cluster are also CLI-only for the same reason: the dashboard shows the cluster, it does not steer it.

Every administrative action is logged with who did it, including the ones that were refused.

What this does not do

  • No email. No approval notice, no self-service password reset, no address verification. The loop is manual on purpose at this size; if it stops being the right size, that is the thing to automate.
  • No SSO. Accounts are local, with local passwords.
  • No roles. There is one flag, is_admin, and it grants looking at the dashboard. Everything else an administrator does is the CLI, and having a shell here is what makes someone one.
  • Rate limits are per gateway instance. With two instances each counts its own, so the effective limit doubles. v1 is single-instance by design.