diff --git a/BACKLOG.md b/BACKLOG.md index 0562b49..c61dfe0 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -372,22 +372,42 @@ portal is not done — the page shows; the console and ctl change. ## C. Features, ranked by value × feasibility -1. **Audit log and roles (M).** Today "audit" is slog lines +1. **Done (branch `features-c`):** `audit_events`, written by the admin + API middleware and the /admin pages (with each change's result), `ctl + audit`, the console's Audit view; staff roles viewer / operator / + billing / admin. Left: before/after values (the request's fields are + recorded, not the state they replaced); a tenant-admin role that changes + its own organization (managers still only view it); hiding the forms a + viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag plus an org "manager" who can view. Add `audit_events` (who, what, before/after, source) written by the `adminAction`/`tenancyAction` wrappers and by every admin-API mutation; roles viewer / operator / tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor. -2. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`, +2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE, + verified email, roles and memberships from groups, `no_passwords`) and + SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is + signed with a per-process key, so with two control planes behind a + balancer the callback must reach the one that started it (sticky + sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`, `internal/web/account.go`; map IdP groups to org/team/admin. -3. **Service accounts and key lifecycle (S).** Per-org maximum key TTL, +3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team, + project), `keys rotate --overlap`, kinds and expiry in `keys list`. + **Service accounts and key lifecycle (S).** Per-org maximum key TTL, `keys rotate` with overlap, last-used shown, project-owned service keys listed apart. `internal/cli/keys.go`, `internal/store/keys.go`, `internal/auth/auth.go`. -4. **Usage export (S).** CSV/Parquet by org/team/project/model/period from +4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl, + console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from the rollups (`internal/store/usageseries.go`), `GET /api/v1/usage/export`, optional scheduled push to S3/SFTP. -5. **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml` +5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node + lost, budget spent, update available; signed JSON or Slack; retries, + hourly de-duplication), `ctl notify test`, Alertmanager receiver + templates. Left: `server init` and the console do not yet ask for a + receiver — the templates are filled in by hand; each control plane sends + budget events on its own, so two behind a balancer may both send one. + **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml` ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver templates driven by `server init` or the console; llmgw-native webhooks for model parked, budget spent, update available (from the reconciler diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ca8b63..cf7b0f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,90 @@ APIs change between minors without a deprecation period. ## Unreleased +### Administration + +- **An audit log.** Every change made through the admin API (`llmgw ctl`, + the console) or the pages under /admin is recorded: who, when, through + what, the fields it was given with passwords, keys and tokens taken out, + and how it ended — refusals included, in words. So are a person's own new + keys, revoked keys and password changes. Read it with `llmgw ctl audit` + (`--since`, `--actor`, `--contains`, `--detail`), `GET /api/v1/audit`, or + the console's **Audit** view. It moves with the store. +- **Staff roles** besides administrator: `viewer` looks at every + administration page and changes nothing, `operator` also runs the cluster + from them, `billing` sees usage and quotas. Set with `llmgw users role` or + the console's **Staff role…**. +- **Single sign-on through OpenID Connect** (`auth.oidc`). A "Sign in + with …" button sends people to the identity provider and back, with PKCE, + state and nonce. The ID token is verified, and the account is found by + the verified email address, or made when `create_accounts` is on. Groups + give staff roles (`roles`, set at every sign-in) and places in the tree + (`memberships`). `no_passwords` leaves the provider as the only way in, + except for administrators. +- **Provisioning through SCIM 2.0** (`auth.scim.token_file`, `/scim/v2`). + - **Users:** the identity provider makes, suspends and reinstates + accounts — the PATCH forms Entra ID and Okta send are both read. + Deleting a user suspends the account. + - **Groups:** joining and leaving the groups `auth.oidc` maps gives and + takes away staff roles and places in the tree. +- **Key lifecycle.** + - **Lifetime limits:** `auth.max_key_ttl` limits how long any key may + live, and `llmgw keys ttl team:Acme/ml 720h` sets shorter limits for an + organization, team or project. A key is made within the shortest that + applies, and one asked for with no expiry is given that longest — from + the CLI, the profile page and the editor connect page alike. + - **Rotation:** `llmgw keys rotate --overlap 48h` issues a + replacement where the old key was, and the old one stops after the + overlap. + - **Listing:** `keys list --all` shows each key's kind — service (a + project's), member, personal — with service keys first, and when it + expires. +- **Usage export.** `llmgw ctl usage export --from 2026-09-01 --to + 2026-10-01 --by project,model` writes CSV by hour, day or month and any of + organization, team, project, user, key and model. + - **Data:** read from the hourly and daily sums, so it reaches back past + the request records' retention. + - **Format:** cost is given in micro-units and units, and names that + look like a spreadsheet formula are quoted. + - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) + and **E** on the console's Usage view. + +### A hardened install by default + +- **`server init` and the setup screens write a hardened configuration.** + - **Guardrails:** a `baseline` policy on every request. It refuses + secrets in prompts and answers, masks card numbers and records injection + attempts. + - **The judge:** Llama Guard 3 1B, two replicas, fetched and run on CPUs + (about half a second for a short text). It refuses violence, crime, + weapons, child abuse, privacy violations, hate, self-harm and sexual + content. The policy fails closed: a request it cannot read within 5 + seconds is refused (503), and `judge_unavailable` goes to the + notification webhooks. + - **Keys:** they live at most 90 days. + - **Opting out:** `--no-guard-model` (or unticking it on the setup screen) + leaves the judge out and keeps the pattern checks. A `server.yaml` + written earlier is never changed. + - **What is left to you:** the closing text now names the two things + init cannot do — a certificate for the client port, and the keys + backups are encrypted to. + +### Notifications + +- **Webhooks for what needs a person** (`notify.webhooks`): a model parked, + a node lost, a budget spent, new weights in a model's repository — sent by + the gateway itself, with no Prometheus in between. + - **Formats:** JSON signed with HMAC-SHA256 (`X-Llmgw-Signature`) when + `secret_file` is set, or a Slack-compatible `{"text": …}`. + - **Delivery:** queued, never holding up what raised it; tried three + times; the same event about the same thing at most once an hour, and new + weights once per revision. + - **Checking:** `llmgw ctl notify test`. + - **A judge not answering** (`judge_unavailable`) is one of the events. +- **Alertmanager receiver templates** for Slack, e-mail, PagerDuty and a + webhook in `deploy/monitoring/alertmanager.yml`, commented out until + chosen. + ### Guardrails - **Hooks for guardrails on the request path** (nothing judges yet; B2 adds diff --git a/README.md b/README.md index 9794ce1..248b969 100644 --- a/README.md +++ b/README.md @@ -8,15 +8,17 @@ OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md). ## Getting started -Starting from nothing — no packages installed, no drivers, no nodes — follow -[docs/install.md](docs/install.md): it builds a control plane, adds a node and -serves a model, and says plainly which parts llmgw does not install for you. +On the machine that will be the control plane, with the release binary +installed: -| Guide | For | -|---|---| -| [docs/install.md](docs/install.md) | Standing up a cluster from scratch | -| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists | -| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys | +```sh +sudo llmgw +``` + +The setup screens install one machine or three (high availability), check +them first, and leave a model answering. After that the same command opens +the console. [docs/quick-install.md](docs/quick-install.md) has the six steps +from empty servers to a first request; everything else is below. ## Status @@ -33,6 +35,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md). | v0.7.0 | — | Conversations, agent memory, disk management | released | | v0.8.0 | — | PostgreSQL, leader election, agent failover | released | | v0.9.0 | — | Managing people and the tenancy tree | released | +| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released | +| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | As of v0.6.0 a cluster runs unattended: agents report their hardware, the @@ -40,12 +44,34 @@ control plane places replicas — including ones larger than a single machine models load when someone asks for them and stop when nobody does, and keys, budgets and rate limits decide who may use any of it. -Before v1.0.0: tracing, reference documentation, and filtering on `ctl nodes` -and `ctl replicas` for fleets too large to read a whole table of. Two items -that were on this list are not gaps: a model falls through to its fallbacks -when there is no capacity as well as when a budget is spent, and routing -profiles are chosen by the key rather than by a request header on purpose — -a header would let any client promote itself into the top priority band. +Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md), +"Unreleased"): + +- **Security hardening** — signed releases only, a token on the admin API, + engines not run as root, TLS for the agent port, etcd and Patroni, a + firewall for llmgw's internal ports, encrypted backups, bounded + connections, sign-in throttling, a Content-Security-Policy. +- **Guardrails** — policies that check requests and answers (PII, secrets, + prompt injection, deny patterns, a judge model such as Llama Guard), in + observe or enforce mode, attached to any part of the tenancy tree: + [docs/guardrails.md](docs/guardrails.md). +- **Administration** — an audit log of every change (`ctl audit`), staff + roles (viewer, operator, billing, admin), single sign-on through OpenID + Connect and provisioning through SCIM 2.0, key lifetime limits and + rotation with overlap (`keys ttl`, `keys rotate`), usage export as CSV + (`ctl usage export`). +- **Notifications** — webhooks, signed or Slack-shaped, for a model parked, + a node lost, a budget spent and new weights published (`notify.webhooks`, + `ctl notify test`), and Alertmanager receiver templates. + +Every one of these is in the console as well as on the command line: +[docs/console.md](docs/console.md). + +What is still ahead is in [BACKLOG.md](BACKLOG.md). Two things that look like +gaps are not: a model falls through to its fallbacks when there is no +capacity as well as when a budget is spent, and routing profiles are chosen +by the key rather than by a request header on purpose — a header would let +any client promote itself into the top priority band. One thing is worth saying plainly: multi-node replicas are proven on a simulated cluster and on real hardware @@ -58,29 +84,50 @@ them. On such a machine, run `llmgw doctor --json` first and keep the report; ## Documentation -- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in - six steps: start here -- [docs/install.md](docs/install.md) — the same from nothing, with every option - explained: control plane, nodes, engines, weights, first request -- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source +Every page is also in the binary: `llmgw docs`, or **Manual** in the console. + +**Installing** + +- [docs/quick-install.md](docs/quick-install.md) — from empty servers to a + first request, in six steps: start here +- [docs/install.md](docs/install.md) — the same with every option explained, + and running the cluster after: monitoring, notifications, single sign-on, + the firewall, upgrades, backups +- [docs/node-onboarding.md](docs/node-onboarding.md) — adding nodes to a + cluster that exists - [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what llmgw needs of them, checking a node with `llmgw doctor`, and what to look at when a replica spanning nodes will not run + +**Running it** + +- [docs/console.md](docs/console.md) — the console that bare `llmgw` opens: + every view, and the command each of its actions runs +- [docs/onboarding.md](docs/onboarding.md) — accounts, keys, limits, staff + roles and single sign-on +- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers: + policies, detectors, a judge model, and where each applies - [docs/choosing-models.md](docs/choosing-models.md) — which models to run, how many of each, and where the weights should live -- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and - what leaves the machine; written for a security review + +**Reference** + - [docs/reference.md](docs/reference.md) — every configuration key, generated from the structs and checked by a test, so it cannot describe a setting that does not exist or miss one that does -- [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest - and most useful thing to build around llmgw -- [deploy/proxy/](deploy/proxy/) — Caddy and nginx in front of the client - port, with streaming and the forwarded headers right -- [deploy/ha/](deploy/ha/) — running two control planes, and the one health - check that has to be right -- [docs/node-onboarding.md](docs/node-onboarding.md), - [docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people +- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and + what leaves the machine; written for a security review +- [deploy/](deploy/) — [a mirror](deploy/mirror/) on its own machine, + [Caddy and nginx](deploy/proxy/) in front of the client port, [two control + planes](deploy/ha/) behind a balancer, and [Prometheus, Alertmanager and + Grafana](deploy/monitoring/) + +**Working on llmgw** + +- [docs/development.md](docs/development.md) — built from source on one + workstation, no systemd and no `sudo` +- [llmgw-plan.md](llmgw-plan.md) — the design; [BACKLOG.md](BACKLOG.md) — + what is next ## Building @@ -100,8 +147,12 @@ llmgw ctl explain-placement llama # why a model has not been placed llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs sudo llmgw doctor # on a node: what its GPUs, NVLink and fabric need +llmgw ctl audit --since 24h # who changed what +llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model ``` +Bare `llmgw` opens the console, which does all of this from one screen. + `ctl` talks to the admin port (`:9090` by default), not the client-facing one. Add `--json` to any command for machine-readable output. diff --git a/deploy/monitoring/alertmanager.yml b/deploy/monitoring/alertmanager.yml index c4ecb02..fbc0360 100644 --- a/deploy/monitoring/alertmanager.yml +++ b/deploy/monitoring/alertmanager.yml @@ -1,10 +1,11 @@ # Alertmanager for the llmgw cluster. # -# Nothing is sent anywhere yet, on purpose. Where an alert should go — an +# Nothing is sent anywhere until a receiver is chosen, on purpose. Where an alert should go — an # address, a Slack workspace, a webhook — is a decision with credentials # attached, and a monitoring system that quietly starts mailing people is worse # than one that is visibly not wired up. Alerts group and fire here and are -# visible at :9093 and in Prometheus; add a receiver below to make them travel. +# visible at :9093 and in Prometheus; uncomment a receiver template below and +# name it on the route to make them travel. global: resolve_timeout: 5m @@ -31,6 +32,53 @@ route: receivers: - name: none + # Receiver templates. Uncomment one, fill in where it goes, and name it in + # `receiver:` above — on the route, the critical branch, or both. Secrets go + # in files Alertmanager reads (the *_file settings), mode 0640 owned by + # root:prometheus, not in this file. `amtool check-config` before a restart. + # + # Slack (an incoming webhook): + # - name: slack + # slack_configs: + # - api_url_file: /etc/prometheus/slack-webhook-url + # channel: '#llmgw' + # send_resolved: true + # title: '{{ .Status | toUpper }}: {{ .CommonLabels.alertname }}' + # text: >- + # {{ range .Alerts }}{{ .Annotations.summary }} + # {{ .Annotations.description }} + # {{ end }} + # + # E-mail (through a relay that takes authenticated submission): + # - name: email + # email_configs: + # - to: ops@example.com + # from: llmgw-alerts@example.com + # smarthost: smtp.example.com:587 + # auth_username: llmgw-alerts@example.com + # auth_password_file: /etc/prometheus/smtp-password + # send_resolved: true + # + # PagerDuty (Events API v2), for the critical branch — what should wake + # someone: + # - name: pagerduty + # pagerduty_configs: + # - routing_key_file: /etc/prometheus/pagerduty-routing-key + # severity: '{{ .CommonLabels.severity }}' + # description: '{{ .CommonAnnotations.summary }}' + # + # A webhook of your own (Alertmanager's JSON, version 4): + # - name: webhook + # webhook_configs: + # - url_file: /etc/prometheus/alert-webhook-url + # send_resolved: true + # max_alerts: 20 + # + # The gateway itself also sends a few events straight to webhooks, with no + # Prometheus in between — a model parked, a node lost, a budget spent, new + # weights published: `notify.webhooks` in server.yaml (docs/install.md, + # "Being told"). + inhibit_rules: # A node that is down explains every replica on it. Reporting both says the # same thing twice and buries whichever is the actual cause. diff --git a/docs/console.md b/docs/console.md index 92e86e2..8a8b047 100644 --- a/docs/console.md +++ b/docs/console.md @@ -67,7 +67,8 @@ and power, and each model's requests a second. The history comes from the Prometheus named by `monitoring.prometheus` in server.yaml; without one the charts draw what this console has seen since it opened, and say so. Network and disk come from each machine's node exporter, which `llmgw node add` -installs. A node's and a model's own pages have their charts too, over the +installs. **N** sends a test notification to each webhook in server.yaml's +`notify` section and says how each answered (`llmgw ctl notify test`). A node's and a model's own pages have their charts too, over the same range. **Nodes** lists every node with its state, load, GPUs, replicas, disk and @@ -158,14 +159,36 @@ project, their limits, a new password (their sessions close), delete. **API keys** — every key, its project, owner, state and last use. **A** issues one; the key is shown once, in the result, and never again. On a row: -its limits, or revoke it. +its limits, **Rotate…** — a new key with the same project, owner and limits, +the old one working on for the overlap you give (`llmgw keys rotate`) — or +revoke it. + +**Accounts** rows also have **Staff role…**: +- `viewer` looks at every page under /admin and changes nothing; +- `operator` also runs the cluster from them (models through the mirror, a + node's engines); +- `billing` sees usage and quotas; +- `admin` may do anything. + +The STAFF column shows each person's role. + +On **Usage**, **E** exports usage as CSV to a file on the control plane: +by hour, day or month and any of organization, team, project, user, key and +model. + +**Audit** lists every change of the last week — through `ctl`, this console +or the /admin pages, and people's own keys and passwords — with who made +it, through what, and how it ended. Enter shows the fields it was given, +with anything secret taken out. `llmgw ctl audit` is the same, with filters. **Organizations** — the tenancy tree: organizations, their teams, the teams' projects, with members, keys and limits. **A** adds an organization. On a row: add a team or project inside it, rename, move, limits for the level and for each of its members, its capacity quota, and for a project how long -conversations and agent notes are kept. **Remove** takes away only something -empty. +conversations and agent notes are kept. **Longest key lifetime…** caps how +long a key made there may live, and is the expiry of one made without any +(`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key). +**Remove** takes away only something empty. Limits are requests and tokens a minute, a budget per day or month, and the models allowed. Each level only narrows what is above it. diff --git a/docs/quickstart.md b/docs/development.md similarity index 98% rename from docs/quickstart.md rename to docs/development.md index 1e8754b..8a8a07c 100644 --- a/docs/quickstart.md +++ b/docs/development.md @@ -1,8 +1,9 @@ -# Quickstart: one machine, one model, ten minutes +# Development: llmgw from source on one machine Written for a developer who wants to see llmgw answer a request on a single -workstation, built from source, with no systemd and no `sudo`. **To install -llmgw on servers, read [quick-install.md](quick-install.md) instead.** +workstation, built from source, with no systemd and no `sudo` — for working on +llmgw itself. **To install llmgw on servers, read +[quick-install.md](quick-install.md) instead.** What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front of a real model served by llama.cpp. diff --git a/docs/guardrails.md b/docs/guardrails.md index ab8c09b..97f471c 100644 --- a/docs/guardrails.md +++ b/docs/guardrails.md @@ -3,8 +3,11 @@ Written for the operator deciding what the gateway lets through: what a policy can check, what it can do, and how one is attached to a team. -Without a `guard:` section in `server.yaml` nothing is checked and nothing -costs anything. With one, every request is judged once it is authorised and +`llmgw server init` writes a `baseline` policy that applies to every +request, with Llama Guard 3 1B as its judge on a CPU; see +[install.md](install.md#2-the-control-plane) for what it checks. Without a +`guard:` section in `server.yaml` nothing is checked and nothing costs +anything. With one, every request is judged once it is authorised and within its limits, with its stored conversation spliced in, and before it is sent to a model; and the answer is judged before the client sees it. diff --git a/docs/install.md b/docs/install.md index 64aa014..9059a9b 100644 --- a/docs/install.md +++ b/docs/install.md @@ -13,34 +13,32 @@ this page says otherwise, and where it says otherwise it says so plainly. Read this first, because it is the difference between an afternoon and a week. -llmgw runs engines and routes requests. It does **not** install GPU drivers, -CUDA, or the inference engines themselves. A node missing any of them still -registers, still reports `ready`, and still fails every replica placed on it — -so the useful order is to make a node able to serve *before* onboarding it, not -after. +llmgw installs itself, its engines and its store. It does **not** install the +GPU driver or the InfiniBand stack: those are kernel modules with version +matrices attached, and a platform decision. A node missing them still +registers and reports `ready`, and fails every replica that needs the GPU or +the fabric — so make a node able to serve *before* onboarding it. +`llmgw doctor` on the node says what is missing and the command that fixes it. | You provide | llmgw provides | | --- | --- | | NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health | | InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one | -| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision | -| Model weights, from somewhere | Fetching them, placement, routing, accounting | +| Nothing — the nodes install them | llama.cpp, vLLM (and SGLang when asked), with the right build for each node's driver and GPU: [Engines](#engines) | +| Which models to run | The mirror that fetches them once, placement, routing, accounting | -The reason is boring and unavoidable: drivers and engines are platform decisions -with kernel modules and version matrices attached. - -Weights are the one item on that list llmgw does handle. It includes a mirror — -step 3 — which holds the files on your own network and hands them to the nodes, -so getting weights onto every machine is not a problem you solve before -starting. What llmgw will not do is choose the models for you. +Weights are handled too. llmgw includes a mirror — step 3 — which holds the +files on your own network and hands them to the nodes, so getting weights onto +every machine is not a problem you solve before starting. What llmgw will not do is choose the models for you. ## 0. Before you begin Four machines is a reasonable first cluster: one control plane and three nodes. One of each also works. -**The control plane** needs no GPU. It is a single process holding a SQLite -file; 2 GB of RAM and a few gigabytes of disk is generous. It must be reachable +**The control plane** needs no GPU: one machine, holding its store in SQLite +or PostgreSQL, or three for high availability, with the store in PostgreSQL replicated under +Patroni. 2 GB of RAM and a few gigabytes of disk each is generous. It must be reachable from every node on the agent port, and from your users on the client port. **Nodes** need whatever the model needs, plus about 8 GB for the operating @@ -67,9 +65,11 @@ a certificate: the machine's own addresses are found for you, and any other name One static binary, no runtime dependencies, the same on every machine. ``` -VERSION=v0.8.0 # check the releases page for the current tag -curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64 -curl -fsSL https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing +VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \ + | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4) +U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION +curl -fsSLO $U/llmgw-$VERSION-linux-amd64 +curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw llmgw --version ``` @@ -117,7 +117,8 @@ That is the whole step. In order, it: 3. issues the control plane's certificate for this machine's hostname and every address it has, and copies `ca.pem` beside it 4. writes `/etc/llmgw/server.yaml`: mutual TLS, authentication, default limits - and a price, and a control plane with an empty catalog + and a price, keys that expire after at most 90 days, the baseline + guardrails policy, and a control plane with an empty catalog 5. creates the store, the `default` project, and your account, already approved and promoted to administrator — the password is generated and **printed once** @@ -127,10 +128,48 @@ That is the whole step. In order, it: 8. makes this machine a node of its own control plane — on its GPUs when `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its engine -9. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the - first thing anyone opens `/chat` to find is a model that answers +9. closes llmgw's internal ports to everything but the cluster's networks +10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas + on the CPUs of nodes with room — this one, or the first that join +11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the + first thing anyone opens `/chat` to find is a model that answers. A + machine without room for it is told so within two minutes, and why. -`--no-mirror`, `--no-node` and `--no-model` leave the last three out. An NVIDIA +`--no-mirror` and `--no-node` leave out 7 and 8, `--no-firewall` 9, +`--no-guard-model` 10 and `--no-model` 11. An NVIDIA card with no working +driver is reported rather than silently ignored; install the driver and run +init again to serve on it. + +**Secure by default.** What init writes is the hardened configuration, not a +demonstration one: + +| | | +|---|---| +| Agents | mutual TLS with the cluster's own authority; the agent port refuses plaintext | +| Admin API | loopback only, behind a token; pprof off | +| Store | between machines, TLS everywhere (PostgreSQL, etcd, Patroni) | +| Firewall | llmgw's internal ports answer the cluster's networks only | +| Accounts | signup closed; administrators create accounts or approve them | +| Keys | at most 90 days (`auth.max_key_ttl`), shorter per part of the tree with `llmgw keys ttl` | +| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind | +| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers | + +The judge runs as two replicas, each on a CPU — about half a second for a +short text on any CPU with AVX2 — and reads a streamed answer every 600 +characters. The policy fails **closed**: a request the judge cannot read +within 5 seconds is refused with 503 rather than served unjudged, and +`judge_unavailable` goes to the notification webhooks. Two replicas are what +make that bearable; on one small machine, or a VM without AVX2 (see +[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the +first thing to fail, so `fail: open` — serve it, recorded as flagged — or +`--no-guard-model` is the choice there. Defamation, specialized advice, +intellectual property and elections are left out of what it refuses: they +catch too much ordinary work. [guardrails.md](guardrails.md) has the rest. + +Two things are left to you because they need something only you have: a +certificate for the client port — port 8080 is plain HTTP; put the proxy in +[deploy/proxy/](../deploy/proxy/) in front — and the keys backups are +encrypted to (`backup_recipients`, see [Backing up](#backing-up)). An NVIDIA card with no working driver is reported rather than silently ignored; install the driver and run init again to serve on it. @@ -927,6 +966,42 @@ fleet means the measurement is missing, not that the fleet is idle. And `llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown, which is what a binary built without the nvml tag reports. +### Being told + +Prometheus alerts travel once `alertmanager.yml` names a receiver; it ships +templates for Slack, e-mail, PagerDuty and a webhook of your own, commented +out until you choose. + +A few things are worth a message even without a monitoring stack, and the +control plane sends those itself: + +| Event | When | +|---|---| +| `model_parked` | A model failed every start in a row and is no longer retried (`ctl retry` tries again). | +| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. | +| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. | +| `update_available` | A model's registry repository published different weights. Once per revision. | +| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. | + +```yaml +notify: + webhooks: + - url: https://hooks.slack.com/services/… # Slack, Mattermost, Rocket.Chat + format: slack + events: [model_parked, node_lost] + - url: https://ops.example.com/llmgw # your own receiver + secret_file: /etc/llmgw/notify-secret # signs the body +``` + +A `json` webhook (the default) gets the event as JSON — `kind`, `at`, +`subject`, `message`, `detail` — with `X-Llmgw-Event` naming the kind and, +when `secret_file` is set, `X-Llmgw-Signature: sha256=`, the HMAC-SHA256 +of the body under the file's contents; check it before believing the body. A +webhook that does not answer is tried three times and then given up on, and the +same event about the same thing is sent at most once an hour, so a budget that +refuses a thousand requests is one message. `llmgw ctl notify test` sends a +test event to each webhook and says how each answered. + ## PostgreSQL, when SQLite is no longer the right shape SQLite is the default and stays the right answer for most installations: one @@ -1066,7 +1141,7 @@ llmgw keys memory --project default --off Without authentication, `threads.store` and `threads.retain_hours` in `server.yaml` keep both conversations and notes in one shared pool. That is for -one trusted machine only; see [quickstart.md](quickstart.md). +one trusted machine only; see [development.md](development.md). ## Two control planes @@ -1080,6 +1155,63 @@ thing to get right is the health check: point it at `/ready`, not `/health`. so a load balancer using it would send half the traffic to something that refuses every request. +## Single sign-on + +People can sign in through your identity provider — Keycloak, Entra ID, +Google, Okta, Authentik, anything that speaks OpenID Connect. + +1. Register a client there with the redirect address + `https:///login/oidc/callback`. +2. Put its secret in a file only root reads. +3. Add to `server.yaml`: + +```yaml +auth: + oidc: + issuer: https://login.example.com/realms/acme + client_id: llmgw + client_secret_file: /etc/llmgw/secrets/oidc-client-secret + label: Acme SSO # the button says "Sign in with Acme SSO" + create_accounts: true # make an active account for someone new + roles: # staff roles from groups, set at each sign-in + admin: [llmgw-admins] + operator: [llmgw-ops] + memberships: # groups to places in the tree, joined at sign-in + ml-engineers: "team:Acme/ml" + no_passwords: true # the provider only; administrators keep a password way in +``` + +How the sign-in works: + +- **Request:** it uses PKCE, a state and a nonce. +- **Identity:** the ID token is verified against the provider's keys, and + the account is found by the verified email address. +- **Unknown addresses:** without `create_accounts`, only people who already + have an account can sign in. +- **Roles:** a person taken out of a group loses the role the next time + they sign in. +- **Memberships:** they are added, never taken away. +- **Audit:** each sign-in is recorded in the audit log. + +### Provisioning (SCIM) + +The provider can also keep accounts in step: made when someone is assigned +the application, suspended when they are removed or leave. Give it a token: + +```yaml +auth: + scim: + token_file: /etc/llmgw/secrets/scim-token # any long random string, root-only +``` + +and point it at `https:///scim/v2` with that token as the bearer. + +- **Users:** the `userName` is the email address. Deleting a user + suspends the account, and its keys and history stay. +- **Groups:** `auth.oidc.roles` and `memberships` are applied as people + join and leave them. Membership in a group is removed when someone leaves + it. + ## The firewall `server init`, `node add`, `server add` and `mirror init` each keep that diff --git a/docs/onboarding.md b/docs/onboarding.md index fbf6672..6662435 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -325,8 +325,15 @@ llmgw keys list --store … --project default minute. It is the column that tells you whether a key is still in use, which is what makes rotation safe: -**To rotate**, issue the new key, let both work while the user switches, watch -`LAST USED` on the old one stop moving, then revoke it. +**To rotate**, `llmgw keys rotate --overlap 48h` issues a new key with +the same project, owner and limits and lets the old one work for the overlap, +then stop. Watch `LAST USED` on the old one stop moving before it runs out. + +**To cap how long keys live**, `llmgw keys ttl project:Acme/ml/train 720h` (or +`team:`, `org:`; `off` removes it), and `auth.max_key_ttl` in server.yaml for +every key. A key is made within the shortest limit that applies, and one asked +for without an expiry gets that limit. `keys list --all` shows each key's kind +— service (a project's), member, personal — and when it expires. ``` llmgw keys revoke --store … key-mpmip77twpt7qaxc @@ -433,21 +440,26 @@ Three things are deliberately not there: | `llmgw keys create` | Issuing a credential on another person's behalf | Each of those turns a stolen admin cookie into more than it was, and each is -rare enough that a shell is no hardship. Cordon, drain and everything else that -touches the cluster are also CLI-only for the same reason: the dashboard shows -the cluster, it does not steer it. +rare enough that a shell is no hardship. Cordon and drain are CLI-only for the +same reason; what the pages do change on the cluster — serving models through +the mirror, a node's engines — needs the `operator` or `admin` staff role. -Every administrative action is logged with who did it, including the ones that -were refused. +Every administrative change is recorded in the audit log — who, when, through +what, the fields given with secrets taken out, and how it ended, refusals +included: `llmgw ctl audit`, or **Audit** in the console. ## What this does not do - **No email.** No approval notice, no self-service password reset, no address verification. The loop is manual on purpose at this size; if it stops being the right size, that is the thing to automate. -- **No SSO.** Accounts are local, with local passwords. -- **No roles.** There is one flag, `is_admin`, and it grants looking at the - dashboard. Everything else an administrator does is the CLI, and having a - shell here is what makes someone one. +- **Single sign-on is optional.** With `auth.oidc` people sign in through + your identity provider, and SCIM 2.0 (`auth.scim`) creates, updates and + suspends accounts from it; see [install.md](install.md#single-sign-on). + Without them accounts are local, with local passwords. +- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin` + (`llmgw users role`) decide what someone may see and do under /admin; an + organization's manager can view it but not change it. Every change is in + the audit log (`llmgw ctl audit`). - **Rate limits are per gateway instance.** With two instances each counts its own, so the effective limit doubles. v1 is single-instance by design. diff --git a/docs/quick-install.md b/docs/quick-install.md index cc77db5..c97037e 100644 --- a/docs/quick-install.md +++ b/docs/quick-install.md @@ -8,10 +8,10 @@ explains every choice. | | | |---|---| -| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. | -| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). | -| **root** | on the control plane, and `sudo` without a password on every node. | -| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. | +| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. | +| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. | +| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. | +| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). | | **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. | You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that @@ -35,26 +35,41 @@ install it. ## 2. Set up the control plane ``` -sudo llmgw server init +sudo llmgw ``` -It asks for your email and a few choices, then sets up everything on this -machine, in a few minutes. In a script, give them as flags instead: -`sudo llmgw server init --admin you@example.com`. +With nothing installed, `llmgw` opens the setup screens. They ask whether +this is **one machine or three** (high availability), and for three, the +other two: address, user, and an ssh key, the ssh agent or a password. They +check every machine before installing anything — system, disk, clock, ports, +internet, whether the machines reach one another — show the plan, and +install: -- the service, its certificates and its configuration; -- your administrator account. **The password is printed once. Copy it now.** +- the service, its certificates and its configuration, and a firewall for + its internal ports; +- with three machines, the store replicated across them (PostgreSQL under + Patroni); +- your administrator account. **The password is shown once. Copy it now.** - a model mirror, which downloads model files once for the whole cluster; -- this machine as a first node, and a small model (Qwen2.5 0.5B) on it. +- this machine as a first node, and a small model (Qwen2.5 0.5B) on it; +- guardrails on every request, with a judge model (Llama Guard 3 1B, two + replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how + to loosen it: [install.md](install.md#2-the-control-plane). + +Run again later, `sudo llmgw` is the console: the cluster's state and +everything you can do to it ([console.md](console.md)). Every step below is +also there, under **Do**. + +**In a script**, the same for one machine: + +``` +sudo llmgw server init --admin you@example.com +``` If nodes will reach this machine by a name or address it does not know itself (a DNS name, a NAT address), add `--host` for each: - -``` -sudo llmgw server init --admin you@example.com --host llm.example.com -``` - -It is safe to run again. It keeps what exists and finishes what is missing. +`--host llm.example.com`. It is safe to run again: it keeps what exists and +finishes what is missing. **The mirror on its own machine**, when it wants a larger disk than the control plane has: first on that machine, with llmgw installed as in step 1, @@ -75,7 +90,7 @@ the account page. ## 3. Add the GPU nodes -On the control plane: +In the console, **Do → Add a node**. Or on the control plane: ``` sudo llmgw node add ubuntu@10.0.0.21 @@ -197,6 +212,9 @@ through the proxy. There is an nginx configuration too: ## Next -- [install.md](install.md): the same steps with every option explained. +- [install.md](install.md): the same steps with every option explained, and + running the cluster after: monitoring, notifications, single sign-on, + backups, upgrades. +- [console.md](console.md): everything the console does. - [fabrics.md](fabrics.md): InfiniBand and NVLink. - [onboarding.md](onboarding.md): accounts, projects, budgets and keys. diff --git a/docs/reference.md b/docs/reference.md index 337a519..df96ec4 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -33,6 +33,9 @@ rather than repeated. | `default_rpm` | `int` | [Auth](#auth) | | `default_tpm` | `int` | [Auth](#auth) | | `signup_open` | `bool` | [Auth](#auth) | +| `oidc` | `OIDC` | [Auth](#auth) | +| `scim` | `SCIM` | [Auth](#auth) | +| `max_key_ttl` | `time.Duration` | [Auth](#auth) | | `personal` | `Personal` | [Auth](#auth) | | `pricing` | `map[string]Price` | [Auth](#auth) | | `default_price` | `Price` | [Auth](#auth) | @@ -106,6 +109,17 @@ rather than repeated. | `prometheus` | `string` | [Monitoring](#monitoring) | | `window` | `time.Duration` | [Monitoring](#monitoring) | | `timeout` | `time.Duration` | [Monitoring](#monitoring) | +| `issuer` | `string` | [OIDC](#oidc) | +| `client_id` | `string` | [OIDC](#oidc) | +| `client_secret_file` | `string` | [OIDC](#oidc) | +| `redirect_url` | `string` | [OIDC](#oidc) | +| `label` | `string` | [OIDC](#oidc) | +| `scopes` | `[]string` | [OIDC](#oidc) | +| `groups_claim` | `string` | [OIDC](#oidc) | +| `create_accounts` | `bool` | [OIDC](#oidc) | +| `roles` | `map[string][]string` | [OIDC](#oidc) | +| `memberships` | `map[string]string` | [OIDC](#oidc) | +| `no_passwords` | `bool` | [OIDC](#oidc) | | `nvidia_smi` | `string` | [Options](#options) | | `ib_root` | `string` | [Options](#options) | | `prefer_nvml` | `bool` | [Options](#options) | @@ -185,6 +199,7 @@ rather than repeated. | `injection` | `string` | [Rules](#rules) | | `deny` | `[]Pattern` | [Rules](#rules) | | `max_chars` | `int` | [Rules](#rules) | +| `token_file` | `string` | [SCIM](#scim) | | `python` | `string` | [SGLangConfig](#sglangconfig) | | `ttft_p99` | `time.Duration` | [SLO](#slo) | | `tpot_p99` | `time.Duration` | [SLO](#slo) | @@ -215,6 +230,7 @@ rather than repeated. | `engine_overrides` | `[]EngineOverride` | [Server](#server) | | `backup_dir` | `string` | [Server](#server) | | `guard` | `guard.Config` | [Server](#server) | +| `notify` | `notify.Config` | [Server](#server) | | `firewall_networks` | `[]string` | [Server](#server) | | `backup_recipients` | `[]string` | [Server](#server) | | `releases` | `string` | [Server](#server) | @@ -241,6 +257,10 @@ rather than repeated. | `bin` | `string` | [VLLMConfig](#vllmconfig) | | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) | | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) | +| `url` | `string` | [Webhook](#webhook) | +| `format` | `string` | [Webhook](#webhook) | +| `events` | `[]string` | [Webhook](#webhook) | +| `secret_file` | `string` | [Webhook](#webhook) | | `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) | | `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) | | `name` | `string` | [admission.Band](#admissionband) | @@ -283,6 +303,8 @@ rather than repeated. | `sim` | `SimConfig` | [engines.Config](#enginesconfig) | | `default` | `[]string` | [guard.Config](#guardconfig) | | `policies` | `map[string]Policy` | [guard.Config](#guardconfig) | +| `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) | +| `host` | `string` | [notify.Config](#notifyconfig) | | `work_dir` | `string` | [supervisor.Config](#supervisorconfig) | | `log_lines` | `int` | [supervisor.Config](#supervisorconfig) | | `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) | @@ -413,6 +435,30 @@ SignupOpen serves the signup form. Accounts it creates are pending until an operator approves them, so this is not a way in — but turning it off means accounts are made with `llmgw users create` instead. +### `oidc` + +`OIDC` + +OIDC is sign-in through an identity provider. + +Takes a [OIDC](#oidc) block. + +### `scim` + +`SCIM` + +SCIM is account provisioning by the identity provider. + +Takes a [SCIM](#scim) block. + +### `max_key_ttl` + +`time.Duration` + +MaxKeyTTL is how long any API key may live; a key asked for with no +expiry is given this. Organizations, teams and projects can set +shorter ones (llmgw keys ttl). Zero is no limit. + ### `personal` `Personal` @@ -959,6 +1005,85 @@ Window is how far back the charts look. Zero takes an hour. Timeout bounds one query. It runs while somebody waits for a page, so a monitoring system that is slow today is skipped rather than waited for. +## OIDC + +### `issuer` + +`string` + +Issuer is the provider's address; its /.well-known/openid-configuration +says the rest. + +### `client_id` + +`string` + +ClientID and the file holding its secret, as registered with the +provider. + +### `client_secret_file` + +`string` + +### `redirect_url` + +`string` + +RedirectURL is where the provider sends people back: +https:///login/oidc/callback. Empty works it out from the +request, which is right unless a proxy changes the address. + +### `label` + +`string` + +Label is what the sign-in button says: "Sign in with