ilker/llmgw
2
0 Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 95 Wiki Activity

Merge features-c: section C of the backlog, items 1-5, and a hardened install
CI / test (push) Successful in 4m13s
Details

Browse Source 
Audit log and staff roles, OIDC and SCIM, key lifecycle, usage export,
notifications; server init writes baseline guardrails with two Llama Guard
judges failing closed, and keys of at most 90 days.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 19:20:41 +02:00
2 parents bcd2413535 c553f2041a
commit e59ff52932
91 files changed
+5161 -180
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

BACKLOG.md
+25 -5
View File
@@ -372,22 +372,42 @@ portal is not done — the page shows; the console and ctl change.
## C. Features, ranked by value × feasibility
1. **Audit log and roles (M).** Today "audit" is slog lines
1. **Done (branch `features-c`):** `audit_events`, written by the admin
API middleware and the /admin pages (with each change's result), `ctl
audit`, the console's Audit view; staff roles viewer / operator /
billing / admin. Left: before/after values (the request's fields are
recorded, not the state they replaced); a tenant-admin role that changes
its own organization (managers still only view it); hiding the forms a
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
plus an org "manager" who can view. Add `audit_events` (who, what,
before/after, source) written by the `adminAction`/`tenancyAction`
wrappers and by every admin-API mutation; roles viewer / operator /
tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
2. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
verified email, roles and memberships from groups, `no_passwords`) and
SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is
signed with a per-process key, so with two control planes behind a
balancer the callback must reach the one that started it (sticky
sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
`internal/web/account.go`; map IdP groups to org/team/admin.
3. **Service accounts and key lifecycle (S).** Per-org maximum key TTL,
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
project), `keys rotate --overlap`, kinds and expiry in `keys list`.
**Service accounts and key lifecycle (S).** Per-org maximum key TTL,
`keys rotate` with overlap, last-used shown, project-owned service keys
listed apart. `internal/cli/keys.go`, `internal/store/keys.go`,
`internal/auth/auth.go`.
4. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
4. **Done (branch `features-c`):** CSV export from the rollups (API, ctl,
console). Left: Parquet, and a scheduled push to S3/SFTP. **Usage export (S).** CSV/Parquet by org/team/project/model/period from
the rollups (`internal/store/usageseries.go`), `GET
/api/v1/usage/export`, optional scheduled push to S3/SFTP.
5. **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
lost, budget spent, update available; signed JSON or Slack; retries,
hourly de-duplication), `ctl notify test`, Alertmanager receiver
templates. Left: `server init` and the console do not yet ask for a
receiver — the templates are filled in by hand; each control plane sends
budget events on its own, so two behind a balancer may both send one.
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
templates driven by `server init` or the console; llmgw-native webhooks
for model parked, budget spent, update available (from the reconciler
CHANGELOG.md
+84
View File
@@ -23,6 +23,90 @@ APIs change between minors without a deprecation period.
## Unreleased
### Administration
- **An audit log.** Every change made through the admin API (`llmgw ctl`,
the console) or the pages under /admin is recorded: who, when, through
what, the fields it was given with passwords, keys and tokens taken out,
and how it ended — refusals included, in words. So are a person's own new
keys, revoked keys and password changes. Read it with `llmgw ctl audit`
(`--since`, `--actor`, `--contains`, `--detail`), `GET /api/v1/audit`, or
the console's **Audit** view. It moves with the store.
- **Staff roles** besides administrator: `viewer` looks at every
administration page and changes nothing, `operator` also runs the cluster
from them, `billing` sees usage and quotas. Set with `llmgw users role` or
the console's **Staff role…**.
- **Single sign-on through OpenID Connect** (`auth.oidc`). A "Sign in
with …" button sends people to the identity provider and back, with PKCE,
state and nonce. The ID token is verified, and the account is found by
the verified email address, or made when `create_accounts` is on. Groups
give staff roles (`roles`, set at every sign-in) and places in the tree
(`memberships`). `no_passwords` leaves the provider as the only way in,
except for administrators.
- **Provisioning through SCIM 2.0** (`auth.scim.token_file`, `/scim/v2`).
- **Users:** the identity provider makes, suspends and reinstates
accounts — the PATCH forms Entra ID and Okta send are both read.
Deleting a user suspends the account.
- **Groups:** joining and leaving the groups `auth.oidc` maps gives and
takes away staff roles and places in the tree.
- **Key lifecycle.**
- **Lifetime limits:** `auth.max_key_ttl` limits how long any key may
live, and `llmgw keys ttl team:Acme/ml 720h` sets shorter limits for an
organization, team or project. A key is made within the shortest that
applies, and one asked for with no expiry is given that longest — from
the CLI, the profile page and the editor connect page alike.
- **Rotation:** `llmgw keys rotate <id> --overlap 48h` issues a
replacement where the old key was, and the old one stops after the
overlap.
- **Listing:** `keys list --all` shows each key's kind — service (a
project's), member, personal — with service keys first, and when it
expires.
- **Usage export.** `llmgw ctl usage export --from 2026-09-01 --to
2026-10-01 --by project,model` writes CSV by hour, day or month and any of
organization, team, project, user, key and model.
- **Data:** read from the hourly and daily sums, so it reaches back past
the request records' retention.
- **Format:** cost is given in micro-units and units, and names that
look like a spreadsheet formula are quoted.
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
and **E** on the console's Usage view.
### A hardened install by default
- **`server init` and the setup screens write a hardened configuration.**
- **Guardrails:** a `baseline` policy on every request. It refuses
secrets in prompts and answers, masks card numbers and records injection
attempts.
- **The judge:** Llama Guard 3 1B, two replicas, fetched and run on CPUs
(about half a second for a short text). It refuses violence, crime,
weapons, child abuse, privacy violations, hate, self-harm and sexual
content. The policy fails closed: a request it cannot read within 5
seconds is refused (503), and `judge_unavailable` goes to the
notification webhooks.
- **Keys:** they live at most 90 days.
- **Opting out:** `--no-guard-model` (or unticking it on the setup screen)
leaves the judge out and keeps the pattern checks. A `server.yaml`
written earlier is never changed.
- **What is left to you:** the closing text now names the two things
init cannot do — a certificate for the client port, and the keys
backups are encrypted to.
### Notifications
- **Webhooks for what needs a person** (`notify.webhooks`): a model parked,
a node lost, a budget spent, new weights in a model's repository — sent by
the gateway itself, with no Prometheus in between.
- **Formats:** JSON signed with HMAC-SHA256 (`X-Llmgw-Signature`) when
`secret_file` is set, or a Slack-compatible `{"text": …}`.
- **Delivery:** queued, never holding up what raised it; tried three
times; the same event about the same thing at most once an hour, and new
weights once per revision.
- **Checking:** `llmgw ctl notify test`.
- **A judge not answering** (`judge_unavailable`) is one of the events.
- **Alertmanager receiver templates** for Slack, e-mail, PagerDuty and a
webhook in `deploy/monitoring/alertmanager.yml`, commented out until
chosen.
### Guardrails
- **Hooks for guardrails on the request path** (nothing judges yet; B2 adds
README.md
+80 -29
View File
@@ -8,15 +8,17 @@ OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
## Getting started
Starting from nothing — no packages installed, no drivers, no nodes — follow
[docs/install.md](docs/install.md): it builds a control plane, adds a node and
serves a model, and says plainly which parts llmgw does not install for you.
On the machine that will be the control plane, with the release binary
installed:
| Guide | For |
|---|---|
| [docs/install.md](docs/install.md) | Standing up a cluster from scratch |
| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists |
| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys |
```sh
sudo llmgw
```
The setup screens install one machine or three (high availability), check
them first, and leave a model answering. After that the same command opens
the console. [docs/quick-install.md](docs/quick-install.md) has the six steps
from empty servers to a first request; everything else is below.
## Status
@@ -33,6 +35,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.7.0 | — | Conversations, agent memory, disk management | released |
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
| v0.9.0 | — | Managing people and the tenancy tree | released |
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -40,12 +44,34 @@ control plane places replicas — including ones larger than a single machine
models load when someone asks for them and stop when nobody does, and keys,
budgets and rate limits decide who may use any of it.
Before v1.0.0: tracing, reference documentation, and filtering on `ctl nodes`
and `ctl replicas` for fleets too large to read a whole table of. Two items
that were on this list are not gaps: a model falls through to its fallbacks
when there is no capacity as well as when a budget is spent, and routing
profiles are chosen by the key rather than by a request header on purpose —
a header would let any client promote itself into the top priority band.
Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md),
"Unreleased"):
- **Security hardening** — signed releases only, a token on the admin API,
engines not run as root, TLS for the agent port, etcd and Patroni, a
firewall for llmgw's internal ports, encrypted backups, bounded
connections, sign-in throttling, a Content-Security-Policy.
- **Guardrails** — policies that check requests and answers (PII, secrets,
prompt injection, deny patterns, a judge model such as Llama Guard), in
observe or enforce mode, attached to any part of the tenancy tree:
[docs/guardrails.md](docs/guardrails.md).
- **Administration** — an audit log of every change (`ctl audit`), staff
roles (viewer, operator, billing, admin), single sign-on through OpenID
Connect and provisioning through SCIM 2.0, key lifetime limits and
rotation with overlap (`keys ttl`, `keys rotate`), usage export as CSV
(`ctl usage export`).
- **Notifications** — webhooks, signed or Slack-shaped, for a model parked,
a node lost, a budget spent and new weights published (`notify.webhooks`,
`ctl notify test`), and Alertmanager receiver templates.
Every one of these is in the console as well as on the command line:
[docs/console.md](docs/console.md).
What is still ahead is in [BACKLOG.md](BACKLOG.md). Two things that look like
gaps are not: a model falls through to its fallbacks when there is no
capacity as well as when a budget is spent, and routing profiles are chosen
by the key rather than by a request header on purpose — a header would let
any client promote itself into the top priority band.
One thing is worth saying plainly:
multi-node replicas are proven on a simulated cluster and on real hardware
@@ -58,29 +84,50 @@ them. On such a machine, run `llmgw doctor --json` first and keep the report;
## Documentation
- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in
six steps: start here
- [docs/install.md](docs/install.md) — the same from nothing, with every option
explained: control plane, nodes, engines, weights, first request
- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source
Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
**Installing**
- [docs/quick-install.md](docs/quick-install.md) — from empty servers to a
first request, in six steps: start here
- [docs/install.md](docs/install.md) — the same with every option explained,
and running the cluster after: monitoring, notifications, single sign-on,
the firewall, upgrades, backups
- [docs/node-onboarding.md](docs/node-onboarding.md) — adding nodes to a
cluster that exists
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
when a replica spanning nodes will not run
**Running it**
- [docs/console.md](docs/console.md) — the console that bare `llmgw` opens:
every view, and the command each of its actions runs
- [docs/onboarding.md](docs/onboarding.md) — accounts, keys, limits, staff
roles and single sign-on
- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
policies, detectors, a judge model, and where each applies
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
how many of each, and where the weights should live
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
what leaves the machine; written for a security review
**Reference**
- [docs/reference.md](docs/reference.md) — every configuration key, generated
from the structs and checked by a test, so it cannot describe a setting that
does not exist or miss one that does
- [deploy/mirror/](deploy/mirror/) — a model mirror, which is the cheapest
and most useful thing to build around llmgw
- [deploy/proxy/](deploy/proxy/) — Caddy and nginx in front of the client
port, with streaming and the forwarded headers right
- [deploy/ha/](deploy/ha/) — running two control planes, and the one health
check that has to be right
- [docs/node-onboarding.md](docs/node-onboarding.md),
[docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
what leaves the machine; written for a security review
- [deploy/](deploy/) — [a mirror](deploy/mirror/) on its own machine,
[Caddy and nginx](deploy/proxy/) in front of the client port, [two control
planes](deploy/ha/) behind a balancer, and [Prometheus, Alertmanager and
Grafana](deploy/monitoring/)
**Working on llmgw**
- [docs/development.md](docs/development.md) — built from source on one
workstation, no systemd and no `sudo`
- [llmgw-plan.md](llmgw-plan.md) — the design; [BACKLOG.md](BACKLOG.md) —
what is next
## Building
@@ -100,8 +147,12 @@ llmgw ctl explain-placement llama # why a model has not been placed
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
sudo llmgw doctor # on a node: what its GPUs, NVLink and fabric need
llmgw ctl audit --since 24h # who changed what
llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model
```
Bare `llmgw` opens the console, which does all of this from one screen.
`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
Add `--json` to any command for machine-readable output.
deploy/monitoring/alertmanager.yml
+50 -2
View File
@@ -1,10 +1,11 @@
# Alertmanager for the llmgw cluster.
#
# Nothing is sent anywhere yet, on purpose. Where an alert should go — an
# Nothing is sent anywhere until a receiver is chosen, on purpose. Where an alert should go — an
# address, a Slack workspace, a webhook — is a decision with credentials
# attached, and a monitoring system that quietly starts mailing people is worse
# than one that is visibly not wired up. Alerts group and fire here and are
# visible at :9093 and in Prometheus; add a receiver below to make them travel.
# visible at :9093 and in Prometheus; uncomment a receiver template below and
# name it on the route to make them travel.
global:
resolve_timeout: 5m
@@ -31,6 +32,53 @@ route:
receivers:
- name: none
# Receiver templates. Uncomment one, fill in where it goes, and name it in
# `receiver:` above — on the route, the critical branch, or both. Secrets go
# in files Alertmanager reads (the *_file settings), mode 0640 owned by
# root:prometheus, not in this file. `amtool check-config` before a restart.
#
# Slack (an incoming webhook):
# - name: slack
# slack_configs:
# - api_url_file: /etc/prometheus/slack-webhook-url
# channel: '#llmgw'
# send_resolved: true
# title: '{{ .Status | toUpper }}: {{ .CommonLabels.alertname }}'
# text: >-
# {{ range .Alerts }}{{ .Annotations.summary }}
# {{ .Annotations.description }}
# {{ end }}
#
# E-mail (through a relay that takes authenticated submission):
# - name: email
# email_configs:
# - to: ops@example.com
# from: llmgw-alerts@example.com
# smarthost: smtp.example.com:587
# auth_username: llmgw-alerts@example.com
# auth_password_file: /etc/prometheus/smtp-password
# send_resolved: true
#
# PagerDuty (Events API v2), for the critical branch — what should wake
# someone:
# - name: pagerduty
# pagerduty_configs:
# - routing_key_file: /etc/prometheus/pagerduty-routing-key
# severity: '{{ .CommonLabels.severity }}'
# description: '{{ .CommonAnnotations.summary }}'
#
# A webhook of your own (Alertmanager's JSON, version 4):
# - name: webhook
# webhook_configs:
# - url_file: /etc/prometheus/alert-webhook-url
# send_resolved: true
# max_alerts: 20
#
# The gateway itself also sends a few events straight to webhooks, with no
# Prometheus in between — a model parked, a node lost, a budget spent, new
# weights published: `notify.webhooks` in server.yaml (docs/install.md,
# "Being told").
inhibit_rules:
# A node that is down explains every replica on it. Reporting both says the
# same thing twice and buries whichever is the actual cause.
docs/console.md
+27 -4
View File
@@ -67,7 +67,8 @@ and power, and each model's requests a second. The history comes from the
Prometheus named by `monitoring.prometheus` in server.yaml; without one the
charts draw what this console has seen since it opened, and say so. Network
and disk come from each machine's node exporter, which `llmgw node add`
installs. A node's and a model's own pages have their charts too, over the
installs. **N** sends a test notification to each webhook in server.yaml's
`notify` section and says how each answered (`llmgw ctl notify test`). A node's and a model's own pages have their charts too, over the
same range.
**Nodes** lists every node with its state, load, GPUs, replicas, disk and
@@ -158,14 +159,36 @@ project, their limits, a new password (their sessions close), delete.
**API keys** — every key, its project, owner, state and last use. **A**
issues one; the key is shown once, in the result, and never again. On a row:
its limits, or revoke it.
its limits, **Rotate…** — a new key with the same project, owner and limits,
the old one working on for the overlap you give (`llmgw keys rotate`) — or
revoke it.
**Accounts** rows also have **Staff role…**:
- `viewer` looks at every page under /admin and changes nothing;
- `operator` also runs the cluster from them (models through the mirror, a
node's engines);
- `billing` sees usage and quotas;
- `admin` may do anything.
The STAFF column shows each person's role.
On **Usage**, **E** exports usage as CSV to a file on the control plane:
by hour, day or month and any of organization, team, project, user, key and
model.
**Audit** lists every change of the last week — through `ctl`, this console
or the /admin pages, and people's own keys and passwords — with who made
it, through what, and how it ended. Enter shows the fields it was given,
with anything secret taken out. `llmgw ctl audit` is the same, with filters.
**Organizations** — the tenancy tree: organizations, their teams, the teams'
projects, with members, keys and limits. **A** adds an organization. On a
row: add a team or project inside it, rename, move, limits for the level and
for each of its members, its capacity quota, and for a project how long
conversations and agent notes are kept. **Remove** takes away only something
empty.
conversations and agent notes are kept. **Longest key lifetime…** caps how
long a key made there may live, and is the expiry of one made without any
(`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key).
**Remove** takes away only something empty.
Limits are requests and tokens a minute, a budget per day or month, and the
models allowed. Each level only narrows what is above it.
docs/quickstart.md → docs/development.md
+4 -3
View File
@@ -1,8 +1,9 @@
# Quickstart: one machine, one model, ten minutes
# Development: llmgw from source on one machine
Written for a developer who wants to see llmgw answer a request on a single
workstation, built from source, with no systemd and no `sudo`. **To install
llmgw on servers, read [quick-install.md](quick-install.md) instead.**
workstation, built from source, with no systemd and no `sudo` — for working on
llmgw itself. **To install llmgw on servers, read
[quick-install.md](quick-install.md) instead.**
What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
of a real model served by llama.cpp.
docs/guardrails.md
+5 -2
View File
@@ -3,8 +3,11 @@
Written for the operator deciding what the gateway lets through: what a
policy can check, what it can do, and how one is attached to a team.
Without a `guard:` section in `server.yaml` nothing is checked and nothing
costs anything. With one, every request is judged once it is authorised and
`llmgw server init` writes a `baseline` policy that applies to every
request, with Llama Guard 3 1B as its judge on a CPU; see
[install.md](install.md#2-the-control-plane) for what it checks. Without a
`guard:` section in `server.yaml` nothing is checked and nothing costs
anything. With one, every request is judged once it is authorised and
within its limits, with its stored conversation spliced in, and before it is
sent to a model; and the answer is judged before the client sees it.
docs/install.md
+156 -24
View File
@@ -13,34 +13,32 @@ this page says otherwise, and where it says otherwise it says so plainly.
Read this first, because it is the difference between an afternoon and a week.
llmgw runs engines and routes requests. It does **not** install GPU drivers,
CUDA, or the inference engines themselves. A node missing any of them still
registers, still reports `ready`, and still fails every replica placed on it —
so the useful order is to make a node able to serve *before* onboarding it, not
after.
llmgw installs itself, its engines and its store. It does **not** install the
GPU driver or the InfiniBand stack: those are kernel modules with version
matrices attached, and a platform decision. A node missing them still
registers and reports `ready`, and fails every replica that needs the GPU or
the fabric — so make a node able to serve *before* onboarding it.
`llmgw doctor` on the node says what is missing and the command that fixes it.
| You provide | llmgw provides |
| --- | --- |
| NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
| InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision |
| Model weights, from somewhere | Fetching them, placement, routing, accounting |
| Nothing — the nodes install them | llama.cpp, vLLM (and SGLang when asked), with the right build for each node's driver and GPU: [Engines](#engines) |
| Which models to run | The mirror that fetches them once, placement, routing, accounting |
The reason is boring and unavoidable: drivers and engines are platform decisions
with kernel modules and version matrices attached.
Weights are the one item on that list llmgw does handle. It includes a mirror —
step 3 — which holds the files on your own network and hands them to the nodes,
so getting weights onto every machine is not a problem you solve before
starting. What llmgw will not do is choose the models for you.
Weights are handled too. llmgw includes a mirror — step 3 — which holds the
files on your own network and hands them to the nodes, so getting weights onto
every machine is not a problem you solve before starting. What llmgw will not do is choose the models for you.
## 0. Before you begin
Four machines is a reasonable first cluster: one control plane and three nodes.
One of each also works.
**The control plane** needs no GPU. It is a single process holding a SQLite
file; 2 GB of RAM and a few gigabytes of disk is generous. It must be reachable
**The control plane** needs no GPU: one machine, holding its store in SQLite
or PostgreSQL, or three for high availability, with the store in PostgreSQL replicated under
Patroni. 2 GB of RAM and a few gigabytes of disk each is generous. It must be reachable
from every node on the agent port, and from your users on the client port.
**Nodes** need whatever the model needs, plus about 8 GB for the operating
@@ -67,9 +65,11 @@ a certificate: the machine's own addresses are found for you, and any other name
One static binary, no runtime dependencies, the same on every machine.
```
VERSION=v0.8.0 # check the releases page for the current tag
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64
curl -fsSL https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
| grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version
```
@@ -117,7 +117,8 @@ That is the whole step. In order, it:
3. issues the control plane's certificate for this machine's hostname and
every address it has, and copies `ca.pem` beside it
4. writes `/etc/llmgw/server.yaml`: mutual TLS, authentication, default limits
and a price, and a control plane with an empty catalog
and a price, keys that expire after at most 90 days, the baseline
guardrails policy, and a control plane with an empty catalog
5. creates the store, the `default` project, and your account, already approved
and promoted to administrator — the password is generated and **printed
once**
@@ -127,10 +128,48 @@ That is the whole step. In order, it:
8. makes this machine a node of its own control plane — on its GPUs when
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
engine
9. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
first thing anyone opens `/chat` to find is a model that answers
9. closes llmgw's internal ports to everything but the cluster's networks
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas
on the CPUs of nodes with room — this one, or the first that join
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
first thing anyone opens `/chat` to find is a model that answers. A
machine without room for it is told so within two minutes, and why.
`--no-mirror`, `--no-node` and `--no-model` leave the last three out. An NVIDIA
`--no-mirror` and `--no-node` leave out 7 and 8, `--no-firewall` 9,
`--no-guard-model` 10 and `--no-model` 11. An NVIDIA card with no working
driver is reported rather than silently ignored; install the driver and run
init again to serve on it.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:
| | |
|---|---|
| Agents | mutual TLS with the cluster's own authority; the agent port refuses plaintext |
| Admin API | loopback only, behind a token; pprof off |
| Store | between machines, TLS everywhere (PostgreSQL, etcd, Patroni) |
| Firewall | llmgw's internal ports answer the cluster's networks only |
| Accounts | signup closed; administrators create accounts or approve them |
| Keys | at most 90 days (`auth.max_key_ttl`), shorter per part of the tree with `llmgw keys ttl` |
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
The judge runs as two replicas, each on a CPU — about half a second for a
short text on any CPU with AVX2 — and reads a streamed answer every 600
characters. The policy fails **closed**: a request the judge cannot read
within 5 seconds is refused with 503 rather than served unjudged, and
`judge_unavailable` goes to the notification webhooks. Two replicas are what
make that bearable; on one small machine, or a VM without AVX2 (see
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
`--no-guard-model` is the choice there. Defamation, specialized advice,
intellectual property and elections are left out of what it refuses: they
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
Two things are left to you because they need something only you have: a
certificate for the client port — port 8080 is plain HTTP; put the proxy in
[deploy/proxy/](../deploy/proxy/) in front — and the keys backups are
encrypted to (`backup_recipients`, see [Backing up](#backing-up)). An NVIDIA
card with no working driver is reported rather than silently ignored; install
the driver and run init again to serve on it.
@@ -927,6 +966,42 @@ fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.
### Being told
Prometheus alerts travel once `alertmanager.yml` names a receiver; it ships
templates for Slack, e-mail, PagerDuty and a webhook of your own, commented
out until you choose.
A few things are worth a message even without a monitoring stack, and the
control plane sends those itself:
| Event | When |
|---|---|
| `model_parked` | A model failed every start in a row and is no longer retried (`ctl retry` tries again). |
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
| `update_available` | A model's registry repository published different weights. Once per revision. |
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |
```yaml
notify:
webhooks:
- url: https://hooks.slack.com/services/… # Slack, Mattermost, Rocket.Chat
format: slack
events: [model_parked, node_lost]
- url: https://ops.example.com/llmgw # your own receiver
secret_file: /etc/llmgw/notify-secret # signs the body
```
A `json` webhook (the default) gets the event as JSON — `kind`, `at`,
`subject`, `message`, `detail` — with `X-Llmgw-Event` naming the kind and,
when `secret_file` is set, `X-Llmgw-Signature: sha256=<hex>`, the HMAC-SHA256
of the body under the file's contents; check it before believing the body. A
webhook that does not answer is tried three times and then given up on, and the
same event about the same thing is sent at most once an hour, so a budget that
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
test event to each webhook and says how each answered.
## PostgreSQL, when SQLite is no longer the right shape
SQLite is the default and stays the right answer for most installations: one
@@ -1066,7 +1141,7 @@ llmgw keys memory --project default --off
Without authentication, `threads.store` and `threads.retain_hours` in
`server.yaml` keep both conversations and notes in one shared pool. That is for
one trusted machine only; see [quickstart.md](quickstart.md).
one trusted machine only; see [development.md](development.md).
## Two control planes
@@ -1080,6 +1155,63 @@ thing to get right is the health check: point it at `/ready`, not `/health`.
so a load balancer using it would send half the traffic to something that
refuses every request.
## Single sign-on
People can sign in through your identity provider — Keycloak, Entra ID,
Google, Okta, Authentik, anything that speaks OpenID Connect.
1. Register a client there with the redirect address
`https://<gateway>/login/oidc/callback`.
2. Put its secret in a file only root reads.
3. Add to `server.yaml`:
```yaml
auth:
oidc:
issuer: https://login.example.com/realms/acme
client_id: llmgw
client_secret_file: /etc/llmgw/secrets/oidc-client-secret
label: Acme SSO # the button says "Sign in with Acme SSO"
create_accounts: true # make an active account for someone new
roles: # staff roles from groups, set at each sign-in
admin: [llmgw-admins]
operator: [llmgw-ops]
memberships: # groups to places in the tree, joined at sign-in
ml-engineers: "team:Acme/ml"
no_passwords: true # the provider only; administrators keep a password way in
```
How the sign-in works:
- **Request:** it uses PKCE, a state and a nonce.
- **Identity:** the ID token is verified against the provider's keys, and
the account is found by the verified email address.
- **Unknown addresses:** without `create_accounts`, only people who already
have an account can sign in.
- **Roles:** a person taken out of a group loses the role the next time
they sign in.
- **Memberships:** they are added, never taken away.
- **Audit:** each sign-in is recorded in the audit log.
### Provisioning (SCIM)
The provider can also keep accounts in step: made when someone is assigned
the application, suspended when they are removed or leave. Give it a token:
```yaml
auth:
scim:
token_file: /etc/llmgw/secrets/scim-token # any long random string, root-only
```
and point it at `https://<gateway>/scim/v2` with that token as the bearer.
- **Users:** the `userName` is the email address. Deleting a user
suspends the account, and its keys and history stay.
- **Groups:** `auth.oidc.roles` and `memberships` are applied as people
join and leave them. Membership in a group is removed when someone leaves
it.
## The firewall
`server init`, `node add`, `server add` and `mirror init` each keep that
docs/onboarding.md
+23 -11
View File
@@ -325,8 +325,15 @@ llmgw keys list --store … --project default
minute. It is the column that tells you whether a key is still in use, which is
what makes rotation safe:
**To rotate**, issue the new key, let both work while the user switches, watch
`LAST USED` on the old one stop moving, then revoke it.
**To rotate**, `llmgw keys rotate <key-id> --overlap 48h` issues a new key with
the same project, owner and limits and lets the old one work for the overlap,
then stop. Watch `LAST USED` on the old one stop moving before it runs out.
**To cap how long keys live**, `llmgw keys ttl project:Acme/ml/train 720h` (or
`team:`, `org:`; `off` removes it), and `auth.max_key_ttl` in server.yaml for
every key. A key is made within the shortest limit that applies, and one asked
for without an expiry gets that limit. `keys list --all` shows each key's kind
— service (a project's), member, personal — and when it expires.
```
llmgw keys revoke --store … key-mpmip77twpt7qaxc
@@ -433,21 +440,26 @@ Three things are deliberately not there:
| `llmgw keys create` | Issuing a credential on another person's behalf |
Each of those turns a stolen admin cookie into more than it was, and each is
rare enough that a shell is no hardship. Cordon, drain and everything else that
touches the cluster are also CLI-only for the same reason: the dashboard shows
the cluster, it does not steer it.
rare enough that a shell is no hardship. Cordon and drain are CLI-only for the
same reason; what the pages do change on the cluster — serving models through
the mirror, a node's engines — needs the `operator` or `admin` staff role.
Every administrative action is logged with who did it, including the ones that
were refused.
Every administrative change is recorded in the audit log — who, when, through
what, the fields given with secrets taken out, and how it ended, refusals
included: `llmgw ctl audit`, or **Audit** in the console.
## What this does not do
- **No email.** No approval notice, no self-service password reset, no address
verification. The loop is manual on purpose at this size; if it stops being
the right size, that is the thing to automate.
- **No SSO.** Accounts are local, with local passwords.
- **No roles.** There is one flag, `is_admin`, and it grants looking at the
dashboard. Everything else an administrator does is the CLI, and having a
shell here is what makes someone one.
- **Single sign-on is optional.** With `auth.oidc` people sign in through
your identity provider, and SCIM 2.0 (`auth.scim`) creates, updates and
suspends accounts from it; see [install.md](install.md#single-sign-on).
Without them accounts are local, with local passwords.
- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
(`llmgw users role`) decide what someone may see and do under /admin; an
organization's manager can view it but not change it. Every change is in
the audit log (`llmgw ctl audit`).
- **Rate limits are per gateway instance.** With two instances each counts its own,
so the effective limit doubles. v1 is single-instance by design.
docs/quick-install.md
+37 -19
View File
@@ -8,10 +8,10 @@ explains every choice.
| | |
|---|---|
| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
| **root** | on the control plane, and `sudo` without a password on every node. |
| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. |
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
@@ -35,26 +35,41 @@ install it.
## 2. Set up the control plane
```
sudo llmgw server init
sudo llmgw
```
It asks for your email and a few choices, then sets up everything on this
machine, in a few minutes. In a script, give them as flags instead:
`sudo llmgw server init --admin you@example.com`.
With nothing installed, `llmgw` opens the setup screens. They ask whether
this is **one machine or three** (high availability), and for three, the
other two: address, user, and an ssh key, the ssh agent or a password. They
check every machine before installing anything — system, disk, clock, ports,
internet, whether the machines reach one another — show the plan, and
install:
- the service, its certificates and its configuration;
- your administrator account. **The password is printed once. Copy it now.**
- the service, its certificates and its configuration, and a firewall for
its internal ports;
- with three machines, the store replicated across them (PostgreSQL under
Patroni);
- your administrator account. **The password is shown once. Copy it now.**
- a model mirror, which downloads model files once for the whole cluster;
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
- guardrails on every request, with a judge model (Llama Guard 3 1B, two
replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how
to loosen it: [install.md](install.md#2-the-control-plane).
Run again later, `sudo llmgw` is the console: the cluster's state and
everything you can do to it ([console.md](console.md)). Every step below is
also there, under **Do**.
**In a script**, the same for one machine:
```
sudo llmgw server init --admin you@example.com
```
If nodes will reach this machine by a name or address it does not know itself
(a DNS name, a NAT address), add `--host` for each:
```
sudo llmgw server init --admin you@example.com --host llm.example.com
```
It is safe to run again. It keeps what exists and finishes what is missing.
`--host llm.example.com`. It is safe to run again: it keeps what exists and
finishes what is missing.
**The mirror on its own machine**, when it wants a larger disk than the
control plane has: first on that machine, with llmgw installed as in step 1,
@@ -75,7 +90,7 @@ the account page.
## 3. Add the GPU nodes
On the control plane:
In the console, **Do → Add a node**. Or on the control plane:
```
sudo llmgw node add ubuntu@10.0.0.21
@@ -197,6 +212,9 @@ through the proxy. There is an nginx configuration too:
## Next
- [install.md](install.md): the same steps with every option explained.
- [install.md](install.md): the same steps with every option explained, and
running the cluster after: monitoring, notifications, single sign-on,
backups, upgrades.
- [console.md](console.md): everything the console does.
- [fabrics.md](fabrics.md): InfiniBand and NVLink.
- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.
docs/reference.md
+183
View File
@@ -33,6 +33,9 @@ rather than repeated.
| `default_rpm` | `int` | [Auth](#auth) |
| `default_tpm` | `int` | [Auth](#auth) |
| `signup_open` | `bool` | [Auth](#auth) |
| `oidc` | `OIDC` | [Auth](#auth) |
| `scim` | `SCIM` | [Auth](#auth) |
| `max_key_ttl` | `time.Duration` | [Auth](#auth) |
| `personal` | `Personal` | [Auth](#auth) |
| `pricing` | `map[string]Price` | [Auth](#auth) |
| `default_price` | `Price` | [Auth](#auth) |
@@ -106,6 +109,17 @@ rather than repeated.
| `prometheus` | `string` | [Monitoring](#monitoring) |
| `window` | `time.Duration` | [Monitoring](#monitoring) |
| `timeout` | `time.Duration` | [Monitoring](#monitoring) |
| `issuer` | `string` | [OIDC](#oidc) |
| `client_id` | `string` | [OIDC](#oidc) |
| `client_secret_file` | `string` | [OIDC](#oidc) |
| `redirect_url` | `string` | [OIDC](#oidc) |
| `label` | `string` | [OIDC](#oidc) |
| `scopes` | `[]string` | [OIDC](#oidc) |
| `groups_claim` | `string` | [OIDC](#oidc) |
| `create_accounts` | `bool` | [OIDC](#oidc) |
| `roles` | `map[string][]string` | [OIDC](#oidc) |
| `memberships` | `map[string]string` | [OIDC](#oidc) |
| `no_passwords` | `bool` | [OIDC](#oidc) |
| `nvidia_smi` | `string` | [Options](#options) |
| `ib_root` | `string` | [Options](#options) |
| `prefer_nvml` | `bool` | [Options](#options) |
@@ -185,6 +199,7 @@ rather than repeated.
| `injection` | `string` | [Rules](#rules) |
| `deny` | `[]Pattern` | [Rules](#rules) |
| `max_chars` | `int` | [Rules](#rules) |
| `token_file` | `string` | [SCIM](#scim) |
| `python` | `string` | [SGLangConfig](#sglangconfig) |
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
| `tpot_p99` | `time.Duration` | [SLO](#slo) |
@@ -215,6 +230,7 @@ rather than repeated.
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `guard` | `guard.Config` | [Server](#server) |
| `notify` | `notify.Config` | [Server](#server) |
| `firewall_networks` | `[]string` | [Server](#server) |
| `backup_recipients` | `[]string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
@@ -241,6 +257,10 @@ rather than repeated.
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
| `url` | `string` | [Webhook](#webhook) |
| `format` | `string` | [Webhook](#webhook) |
| `events` | `[]string` | [Webhook](#webhook) |
| `secret_file` | `string` | [Webhook](#webhook) |
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
| `name` | `string` | [admission.Band](#admissionband) |
@@ -283,6 +303,8 @@ rather than repeated.
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
| `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) |
| `host` | `string` | [notify.Config](#notifyconfig) |
| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
@@ -413,6 +435,30 @@ SignupOpen serves the signup form. Accounts it creates are pending until
an operator approves them, so this is not a way in — but turning it off
means accounts are made with `llmgw users create` instead.
### `oidc`
`OIDC`
OIDC is sign-in through an identity provider.
Takes a [OIDC](#oidc) block.
### `scim`
`SCIM`
SCIM is account provisioning by the identity provider.
Takes a [SCIM](#scim) block.
### `max_key_ttl`
`time.Duration`
MaxKeyTTL is how long any API key may live; a key asked for with no
expiry is given this. Organizations, teams and projects can set
shorter ones (llmgw keys ttl). Zero is no limit.
### `personal`
`Personal`
@@ -959,6 +1005,85 @@ Window is how far back the charts look. Zero takes an hour.
Timeout bounds one query. It runs while somebody waits for a page, so a
monitoring system that is slow today is skipped rather than waited for.
## OIDC
### `issuer`
`string`
Issuer is the provider's address; its /.well-known/openid-configuration
says the rest.
### `client_id`
`string`
ClientID and the file holding its secret, as registered with the
provider.
### `client_secret_file`
`string`
### `redirect_url`
`string`
RedirectURL is where the provider sends people back:
https://<gateway>/login/oidc/callback. Empty works it out from the
request, which is right unless a proxy changes the address.
### `label`
`string`
Label is what the sign-in button says: "Sign in with <label>".
### `scopes`
`[]string`
Scopes asked for besides openid; empty is email and profile, and
groups when Roles or Memberships use them.
### `groups_claim`
`string`
GroupsClaim names the ID token claim holding groups; empty is
"groups".
### `create_accounts`
`bool`
CreateAccounts makes an account, active, for a person the provider
vouches for and the gateway does not know. Off, only existing accounts
sign in this way.
### `roles`
`map[string][]string`
Roles maps a staff role (viewer, operator, billing, admin) to the
groups that give it. Set at every sign-in: someone taken out of the
group loses the role the next time.
### `memberships`
`map[string]string`
Memberships maps a group to a place in the tree — org:NAME,
team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in.
### `no_passwords`
`bool`
PasswordSignIn off refuses passwords on the sign-in form: people come
in through the provider only. Administrators still can, so a broken
provider does not lock everybody out.
## Options
### `nvidia_smi`
@@ -1492,6 +1617,15 @@ Deny are patterns, matched against the text as a model reads it.
MaxChars refuses text longer than this; zero is no limit.
## SCIM
### `token_file`
`string`
TokenFile holds the bearer token the provider sends. Empty turns
provisioning off.
## SGLangConfig
### `python`
@@ -1741,6 +1875,16 @@ guard.default, and to parts of the tenancy tree by attaching them
Takes a [guard.Config](#guardconfig) block.
### `notify`
`notify.Config`
Notify sends what needs a person — a model parked, a node lost, a
budget spent, new weights published — to webhooks, signed, without a
Prometheus and Alertmanager in between. Empty sends nothing.
Takes a [notify.Config](#notifyconfig) block.
### `firewall_networks`
`[]string`
@@ -1929,6 +2073,33 @@ node and has not been checked against a real cluster.
RayBin is used when LaunchMode is "ray".
## Webhook
### `url`
`string`
### `format`
`string`
Format is json (the event, with X-Llmgw-Signature when SecretFile is
set) or slack ({"text": …}, which Slack, Mattermost and Rocket.Chat
incoming webhooks take).
### `events`
`[]string`
Events are the kinds sent here; empty is all.
### `secret_file`
`string`
SecretFile holds the key the body is signed with: X-Llmgw-Signature
is sha256=<hex HMAC of the body>.
## accounting.Price
### `prompt_per_1k`
@@ -2213,6 +2384,18 @@ Takes a [SimConfig](#simconfig) block.
`map[string]Policy`
## notify.Config
### `webhooks`
`[]Webhook`
### `host`
`string`
Host names the gateway in messages; empty is this machine's name.
## supervisor.Config
### `work_dir`
go.mod
+4 -1
View File
@@ -10,6 +10,8 @@ require (
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/charmbracelet/x/ansi v0.11.6
github.com/coreos/go-oidc/v3 v3.21.0
github.com/go-jose/go-jose/v4 v4.1.4
github.com/jackc/pgx/v5 v5.11.0
github.com/knadh/koanf/parsers/yaml v1.1.1
github.com/knadh/koanf/providers/env/v2 v2.0.1
@@ -28,7 +30,9 @@ require (
go.yaml.in/yaml/v3 v3.0.5
golang.org/x/crypto v0.57.0
golang.org/x/net v0.59.0
golang.org/x/oauth2 v0.37.0
golang.org/x/sync v0.23.0
golang.org/x/text v0.42.0
google.golang.org/protobuf v1.36.12
modernc.org/sqlite v1.59.0
)
@@ -79,7 +83,6 @@ require (
go.opentelemetry.io/otel/metric v1.46.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.42.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/grpc v1.83.1 // indirect
go.sum
+6
View File
@@ -36,6 +36,8 @@ github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfa
github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
github.com/clipperhouse/uax29/v2 v2.5.0 h1:x7T0T4eTHDONxFJsL94uKNKPHrclyFI0lm7+w94cO8U=
github.com/clipperhouse/uax29/v2 v2.5.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
github.com/coreos/go-oidc/v3 v3.21.0 h1:wZo4Q9Pum8dYEj0eMUPrqR+kvuGkeUplbLpNCkBqoWM=
github.com/coreos/go-oidc/v3 v3.21.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26KVTCI9SrY4=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
@@ -44,6 +46,8 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -169,6 +173,8 @@ golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/oauth2 v0.37.0 h1:JUlcxA8oAtauLfiH8FX2/FkAWHAdi0QtGCGc+hofE98=
golang.org/x/oauth2 v0.37.0/go.mod h1:IxwZNxUULJmpBFf9K/9NTMSIfZZuvuTy1gGxhigP/58=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
internal/accounting/accounting.go
+11
View File
@@ -6,9 +6,11 @@ import (
"fmt"
"log/slog"
"strings"
"sync/atomic"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -45,8 +47,13 @@ type Accountant struct {
tokens *Limiter
recorder *Recorder
log *slog.Logger
// notify is told when a budget runs out; nil tells no one.
notify atomic.Pointer[notify.Notifier]
}
// SetNotifier gives the accountant somewhere to say a budget is spent.
func (a *Accountant) SetNotifier(n *notify.Notifier) { a.notify.Store(n) }
// New builds an accountant.
func New(pricing *Pricing, recorder *Recorder, log *slog.Logger) *Accountant {
if log == nil {
@@ -182,6 +189,10 @@ func (a *Accountant) Check(p auth.Principal, model string, estimatedTokens int64
if !price.Zero() {
for _, l := range chain {
if l.Budget != nil && a.budgets.Exhausted(l.Counter, l.Period, *l.Budget) {
a.notify.Load().Notify(notify.BudgetSpent, l.Counter,
fmt.Sprintf("the %s budget of %s is spent: requests to priced models are refused until it renews",
or(string(l.Period), "total"), l.Counter),
map[string]any{"counter": l.Counter, "period": string(l.Period), "budget": *l.Budget})
return res, &LimitError{Err: ErrBudgetSpent}
}
}
internal/accounting/accounting_test.go
+27
View File
@@ -5,11 +5,14 @@ import (
"errors"
"log/slog"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify/notifytest"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -677,3 +680,27 @@ func TestAFallbackIsChargedWhatTheModelThatAnsweredCosts(t *testing.T) {
t.Error("the paid model answering cost nothing")
}
}
func TestASpentBudgetTellsSomeoneOnce(t *testing.T) {
a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
n, sent := notifytest.Catch(t)
a.SetNotifier(n)
p := principal("k1", func(e *store.Effective) {
e.Budget = i64p(100)
e.BudgetPeriod = store.PeriodMonth
})
res, err := a.Check(p, "paid", 10)
if err != nil {
t.Fatal(err)
}
a.Record(res, store.Usage{KeyID: "k1", PromptTokens: 1000})
for range 3 {
if _, err := a.Check(p, "paid", 10); !errors.Is(err, ErrBudgetSpent) {
t.Fatalf("not refused: %v", err)
}
}
got := sent()
if len(got) != 1 || got[0].Kind != notify.BudgetSpent || !strings.Contains(got[0].Message, "month") {
t.Fatalf("sent %+v, want one budget_spent", got)
}
}
internal/audit/audit.go
+152
View File
@@ -0,0 +1,152 @@
// Package audit records who changed what: every change made through the
// admin API and the administration pages, written to the store with the
// request's fields — anything secret taken out — and how it ended.
package audit
import (
"bytes"
"context"
"encoding/json"
"io"
"log/slog"
"net/http"
"net/url"
"regexp"
"sort"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Sink is where events go: the store.
type Sink interface {
WriteAudit(ctx context.Context, e store.AuditEvent) error
}
// secretName is a field whose value is never recorded.
// The password form's fields are current and new.
var secretName = regexp.MustCompile(`(?i)(pass|secret|token|api_?key|^key$|dsn|private|credential|csrf|cookie|^current$|^new$|^confirm)`)
// Redact is a request's fields with secrets replaced, as JSON.
func Redact(fields map[string]any) string {
clean := redactValue(fields)
b, _ := json.Marshal(clean)
if len(b) > 8192 {
b = append(b[:8192], []byte("…")...)
}
return string(b)
}
func redactValue(v any) any {
switch t := v.(type) {
case map[string]any:
out := make(map[string]any, len(t))
for k, val := range t {
if secretName.MatchString(k) {
out[k] = "***"
continue
}
out[k] = redactValue(val)
}
return out
case []any:
out := make([]any, len(t))
for i, val := range t {
out[i] = redactValue(val)
}
return out
case string:
// A secret pasted where a name was expected.
if strings.HasPrefix(t, "sk-") || strings.Contains(t, "PRIVATE KEY") {
return "***"
}
return t
}
return v
}
// FormFields is a form as fields, for Redact.
func FormFields(v url.Values) map[string]any {
out := map[string]any{}
keys := make([]string, 0, len(v))
for k := range v {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
if len(v[k]) == 1 {
out[k] = v[k][0]
} else {
out[k] = v[k]
}
}
return out
}
// statusWriter keeps the status a handler answered with.
type statusWriter struct {
http.ResponseWriter
status int
}
func (w *statusWriter) WriteHeader(code int) {
if w.status == 0 {
w.status = code
}
w.ResponseWriter.WriteHeader(code)
}
func (w *statusWriter) Write(b []byte) (int, error) {
if w.status == 0 {
w.status = http.StatusOK
}
return w.ResponseWriter.Write(b)
}
func (w *statusWriter) Flush() {
if f, ok := w.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
// API records every change to the admin API: a request that is not a GET or
// HEAD, under /api/. actor says who made it.
func API(sink Sink, actor func(*http.Request) string, log *slog.Logger, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet || r.Method == http.MethodHead || !strings.HasPrefix(r.URL.Path, "/api/") {
next.ServeHTTP(w, r)
return
}
var body []byte
if r.Body != nil {
body, _ = io.ReadAll(io.LimitReader(r.Body, 64<<10))
r.Body = io.NopCloser(bytes.NewReader(body))
}
sw := &statusWriter{ResponseWriter: w}
next.ServeHTTP(sw, r)
e := store.AuditEvent{At: time.Now(), Actor: actor(r), Source: "api",
Action: r.Method + " " + r.URL.Path, Status: sw.status}
var fields map[string]any
if json.Unmarshal(body, &fields) == nil {
e.Detail = Redact(fields)
}
for _, k := range []string{"name", "target", "scope", "model"} {
if s, ok := fields[k].(string); ok && e.Target == "" {
e.Target = s
}
}
Write(r.Context(), sink, log, e)
})
}
// Write records an event, saying so in the log when the store will not take
// it: a change that happened and is not in the audit is worth a warning.
func Write(ctx context.Context, sink Sink, log *slog.Logger, e store.AuditEvent) {
if sink == nil {
return
}
if err := sink.WriteAudit(context.WithoutCancel(ctx), e); err != nil && log != nil {
log.Warn("a change could not be written to the audit log", "action", e.Action, "actor", e.Actor, "err", err)
}
}
internal/audit/audit_test.go
+60
View File
@@ -0,0 +1,60 @@
package audit
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func TestNothingSecretIsRecorded(t *testing.T) {
got := Redact(FormFields(url.Values{
"current": {"old-pw"}, "new": {"new-pw"}, "password": {"pw"}, "csrf": {"tok"},
"name": {"ok-name"}, "note": {"sk-proj-abcdefgh"},
}))
for _, secret := range []string{"old-pw", "new-pw", `"pw"`, "tok", "sk-proj"} {
if strings.Contains(got, secret) {
t.Errorf("%s recorded: %s", secret, got)
}
}
if !strings.Contains(got, "ok-name") {
t.Errorf("an ordinary field was dropped: %s", got)
}
nested := Redact(map[string]any{"external": map[string]any{"api_key": "x1", "url": "https://p"}})
if strings.Contains(nested, "x1") || !strings.Contains(nested, "https://p") {
t.Errorf("nested: %s", nested)
}
}
type memSink struct{ got []store.AuditEvent }
func (m *memSink) WriteAudit(_ context.Context, e store.AuditEvent) error {
m.got = append(m.got, e)
return nil
}
func TestEveryChangeToTheAdminAPIIsRecorded(t *testing.T) {
sink := &memSink{}
h := API(sink, func(*http.Request) string { return "ilker" }, nil, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusCreated)
}))
for _, req := range []*http.Request{
httptest.NewRequest("GET", "/api/v1/models", nil),
httptest.NewRequest("POST", "/api/v1/models", strings.NewReader(`{"name":"m1","external":{"api_key":"sk-1"}}`)),
httptest.NewRequest("DELETE", "/api/v1/nodes/n1", nil),
} {
h.ServeHTTP(httptest.NewRecorder(), req)
}
if len(sink.got) != 2 {
t.Fatalf("recorded %d: %+v", len(sink.got), sink.got)
}
e := sink.got[0]
if e.Actor != "ilker" || e.Action != "POST /api/v1/models" || e.Target != "m1" || e.Status != 201 ||
strings.Contains(e.Detail, "sk-1") || e.Source != "api" {
t.Errorf("%+v", e)
}
}
internal/cli/audit.go
+67
View File
@@ -0,0 +1,67 @@
package cli
import (
"encoding/json"
"fmt"
"net/url"
"strconv"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func newAuditCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
var (
since, actor, contains string
limit int
detail bool
)
c := &cobra.Command{
Use: "audit",
Short: "Who changed what: every change through the admin API and the administration pages",
Long: "Each change made with `llmgw ctl`, the console, or the pages under /admin —\n" +
"and a person's own keys and password — is recorded: who, when, through\n" +
"what, the fields it was given with anything secret taken out, and how it\n" +
"ended. The newest first.",
Example: " llmgw ctl audit --since 24h\n llmgw ctl audit --actor ilker --contains models\n llmgw ctl audit --contains bob@example.com --detail",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
q := url.Values{"since": {since}, "limit": {strconv.Itoa(limit)}}
if actor != "" {
q.Set("actor", actor)
}
if contains != "" {
q.Set("contains", contains)
}
var ev []store.AuditEvent
if err := client().get(cmd.Context(), "/api/v1/audit?"+q.Encode(), &ev); err != nil {
return err
}
if *asJSON {
return json.NewEncoder(cmd.OutOrStdout()).Encode(ev)
}
if len(ev) == 0 {
fmt.Fprintln(cmd.OutOrStdout(), "No changes recorded in that time.")
return nil
}
w := tab()
fmt.Fprintln(w, "WHEN (UTC)\tWHO\tVIA\tWHAT\tTARGET\tSTATUS")
for _, e := range ev {
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%d\n", e.At.UTC().Format("2006-01-02 15:04:05"),
e.Actor, e.Source, e.Action, dash(e.Target), e.Status)
if detail && e.Detail != "" && e.Detail != "{}" {
fmt.Fprintf(w, "\t\t\t %s\t\t\n", e.Detail)
}
}
return w.Flush()
},
}
f := c.Flags()
f.StringVar(&since, "since", "7d", "how far back: 90m, 24h, 30d")
f.StringVar(&actor, "actor", "", "only changes by this person")
f.StringVar(&contains, "contains", "", "only changes whose action or target contains this")
f.IntVar(&limit, "limit", 200, "at most this many")
f.BoolVar(&detail, "detail", false, "show the fields each change was given")
return c
}
internal/cli/ctl.go
+2
View File
@@ -161,6 +161,8 @@ func newCtlCmd() *cobra.Command {
newCapacityCmd(client, &asJSON),
newQuotaCmd(client, &asJSON),
newGuardCmd(client, &asJSON),
newAuditCmd(client, &asJSON),
newNotifyCmd(client, &asJSON),
newUsageCmd(client, &asJSON),
newCertCmd(),
)
internal/cli/keys.go
+175 -3
View File
@@ -7,6 +7,7 @@ import (
"errors"
"fmt"
"math"
"slices"
"strings"
"text/tabwriter"
"time"
@@ -37,6 +38,8 @@ func newTenancyCmd() *cobra.Command {
newKeyCreateCmd(open),
newKeyListCmd(open),
newKeyRevokeCmd(open),
newKeyRotateCmd(open),
newKeyTTLCmd(open),
newLimitsCmd(open),
newThreadsCmd(open),
newMemoryCmd(open),
@@ -220,15 +223,20 @@ func newKeyListCmd(open opener) *cobra.Command {
return err
}
w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
fmt.Fprintln(w, "ID\tNAME\tKEY\tPROJECT\tOWNER\tSTATE\tLAST USED")
fmt.Fprintln(w, "ID\tNAME\tKIND\tKEY\tPROJECT\tOWNER\tSTATE\tEXPIRES\tLAST USED")
now := time.Now()
// Service keys — a project's, nobody's own — first: they are
// the ones nobody leaving the company takes with them.
slices.SortStableFunc(rows, func(a, b store.KeyRow) int {
return strings.Compare(keyKind(a.Key), keyKind(b.Key))
})
for _, r := range rows {
where := r.ProjectPath
if where == "" {
where = "personal"
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", r.ID, r.Name, r.Prefix,
where, dash(r.OwnerEmail), keyState(r.Key, now), ago(r.LastUsedAt, now))
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", r.ID, r.Name, keyKind(r.Key), r.Prefix,
where, dash(r.OwnerEmail), keyState(r.Key, now), until(r.ExpiresAt, now), ago(r.LastUsedAt, now))
}
return w.Flush()
}
@@ -387,6 +395,33 @@ func newLimitsCmd(open opener) *cobra.Command {
return cmd
}
// keyKind says whose a key is: a project's (service), a person's in a
// project (member), or a person's own (personal).
func keyKind(k store.Key) string {
switch {
case k.UserID == "":
return "service"
case k.ProjectID == "":
return "personal"
}
return "member"
}
// until is how long a key has left, or "never".
func until(t, now time.Time) string {
if t.IsZero() {
return "never"
}
if !t.After(now) {
return "-"
}
d := t.Sub(now)
if d > 48*time.Hour {
return fmt.Sprintf("in %dd", int(d.Hours()/24))
}
return "in " + d.Round(time.Minute).String()
}
func keyState(k store.Key, now time.Time) string {
switch {
case !k.RevokedAt.IsZero():
@@ -427,3 +462,140 @@ func randomID(prefix string) string {
_, _ = rand.Read(buf[:])
return prefix + "-" + strings.ToLower(base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(buf[:]))
}
func newKeyRotateCmd(open opener) *cobra.Command {
var overlap time.Duration
cmd := &cobra.Command{
Use: "rotate <key-id>",
Short: "Replace a key with a new one, the old one working on for a while",
Long: "The new key belongs where the old one did — the same project, owner, band\n" +
"and routing profile, and as long a life — and is printed once. The old\n" +
"one stops working after --overlap, so whatever uses it can be moved over\n" +
"without an outage.",
Example: " llmgw keys rotate key-7f3a --overlap 48h",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
s, err := open(cmd.Context())
if err != nil {
return err
}
defer s.Close()
old, err := s.KeyByID(cmd.Context(), args[0])
if err != nil {
return fmt.Errorf("no key %s: %w", args[0], err)
}
if !old.RevokedAt.IsZero() {
return fmt.Errorf("%s is revoked; make a new key instead", old.ID)
}
secret, k, err := store.NewKey(randomID("key"), old.Name, old.ProjectID)
if err != nil {
return err
}
k.CreatedAt = time.Now()
k.Band, k.RoutingProfile, k.UserID = old.Band, old.RoutingProfile, old.UserID
if !old.ExpiresAt.IsZero() {
k.ExpiresAt = k.CreatedAt.Add(old.ExpiresAt.Sub(old.CreatedAt))
}
if err := s.CreateKey(cmd.Context(), k); err != nil {
return err
}
retire := time.Now().Add(overlap)
if err := s.ExpireKeyBy(cmd.Context(), old.ID, retire); err != nil {
return err
}
fmt.Fprintln(cmd.OutOrStdout(), secret)
cmd.PrintErrf("\nid %s (replaces %s)\nold key stops working at %s\n", k.ID, old.ID,
retire.UTC().Format(time.RFC3339))
return nil
},
}
cmd.Flags().DurationVar(&overlap, "overlap", 24*time.Hour, "how long the old key keeps working")
return cmd
}
func newKeyTTLCmd(open opener) *cobra.Command {
return &cobra.Command{
Use: "ttl [<org|team|project>:<name> <duration|off>]",
Short: "How long a key may live at most, in a part of the tree",
Long: "A key is made within the shortest limit that applies to it — its\n" +
"project's, its team's, its organization's, and auth.max_key_ttl in\n" +
"server.yaml for every key. A key asked for with no expiry is given the\n" +
"longest allowed. Existing keys are not changed. Without arguments, lists\n" +
"the limits.",
Example: " llmgw keys ttl team:Acme/ml 720h\n llmgw keys ttl team:Acme/ml off",
Args: cobra.RangeArgs(0, 2),
RunE: func(cmd *cobra.Command, args []string) error {
s, err := open(cmd.Context())
if err != nil {
return err
}
defer s.Close()
if len(args) == 0 {
limits, err := s.KeyTTLs(cmd.Context())
if err != nil {
return err
}
if len(limits) == 0 {
fmt.Fprintln(cmd.OutOrStdout(), "No limit: keys may live for ever.")
return nil
}
names := make([]string, 0, len(limits))
for k := range limits {
names = append(names, k)
}
slices.Sort(names)
w := tabwriter.NewWriter(cmd.OutOrStdout(), 0, 0, 2, ' ', 0)
fmt.Fprintln(w, "SCOPE\tLONGEST")
for _, n := range names {
fmt.Fprintf(w, "%s\t%s\n", strings.TrimSuffix(n, ":"), limits[n])
}
return w.Flush()
}
if len(args) != 2 {
return errors.New("give a scope and a duration, or off")
}
kind, name, ok := strings.Cut(args[0], ":")
if !ok {
return fmt.Errorf("%q is not org:NAME, team:NAME or project:NAME", args[0])
}
var id string
switch store.Scope(kind) {
case store.ScopeOrg:
o, err := resolveOrg(cmd.Context(), s, name)
if err != nil {
return err
}
id = o.ID
case store.ScopeTeam:
t, err := resolveTeam(cmd.Context(), s, name)
if err != nil {
return err
}
id = t.ID
case store.ScopeProject:
p, err := resolveProject(cmd.Context(), s, name)
if err != nil {
return err
}
id = p.ID
default:
return fmt.Errorf("%q: org, team or project — the gateway's is auth.max_key_ttl in server.yaml", kind)
}
var d time.Duration
if args[1] != "off" {
if d, err = time.ParseDuration(args[1]); err != nil || d <= 0 {
return fmt.Errorf("%q is not a duration like 720h, or off", args[1])
}
}
if err := s.SetMaxKeyTTL(cmd.Context(), store.Scope(kind), id, d); err != nil {
return err
}
if d == 0 {
fmt.Fprintf(cmd.OutOrStdout(), "%s: no limit of its own\n", args[0])
} else {
fmt.Fprintf(cmd.OutOrStdout(), "%s: keys live %s at most\n", args[0], d)
}
return nil
},
}
}
internal/cli/keys_lifecycle_test.go
+51
View File
@@ -0,0 +1,51 @@
package cli
import (
"path/filepath"
"regexp"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func TestAKeyIsRotatedWithAnOverlapAndKeysLiveWithinTheirLimit(t *testing.T) {
db := filepath.Join(t.TempDir(), "s.db")
mustRun(t, db, "org", "add", "org", "Acme")
mustRun(t, db, "org", "add", "team", "ml", "--in", "Acme")
mustRun(t, db, "org", "add", "project", "train", "--in", "Acme/ml")
mustRun(t, db, "keys", "ttl", "team:Acme/ml", "720h")
if out := mustRun(t, db, "keys", "ttl"); !strings.Contains(out, "720h") {
t.Errorf("listing: %s", out)
}
if _, err := run(t, db, "keys", "create", "ci", "--project", "train", "--expires-in", "2000h"); err == nil {
t.Error("a key past the team's limit was made")
}
mustRun(t, db, "keys", "create", "ci", "--project", "train")
list := mustRun(t, db, "keys", "list", "--all")
if !strings.Contains(list, "service") || !strings.Contains(list, "in 29d") && !strings.Contains(list, "in 30d") {
t.Errorf("a service key, given the team's longest life:\n%s", list)
}
id := regexp.MustCompile(`key-[a-z0-9]+`).FindString(list)
out := mustRun(t, db, "keys", "rotate", id, "--overlap", "2h")
if !strings.HasPrefix(strings.TrimSpace(out), store.KeyPrefix) {
t.Errorf("the new key was not printed: %q", out)
}
s, err := store.Open(t.Context(), db)
if err != nil {
t.Fatal(err)
}
defer s.Close()
old, _ := s.KeyByID(t.Context(), id)
if d := time.Until(old.ExpiresAt); d > 2*time.Hour+time.Minute || d < time.Hour {
t.Errorf("the old key stops in %s, expected about 2h", d)
}
rows, _ := s.AllKeys(t.Context())
if len(rows) != 2 {
t.Errorf("%d keys after rotating", len(rows))
}
}
internal/cli/notify.go
+48
View File
@@ -0,0 +1,48 @@
package cli
import (
"encoding/json"
"fmt"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
)
func newNotifyCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
c := &cobra.Command{
Use: "notify",
Short: "The webhooks told about parked models, lost nodes, spent budgets and new weights",
}
c.AddCommand(&cobra.Command{
Use: "test",
Short: "Send a test notification to every webhook in server.yaml's notify section",
Long: "Sends one event of kind \"test\" to each webhook, once, and says how each\n" +
"answered: the way to check an address, a secret and a firewall before\n" +
"something real has to travel.",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var res []notify.Result
if err := client().post(cmd.Context(), "/api/v1/notify/test", nil, &res); err != nil {
return err
}
if *asJSON {
return json.NewEncoder(cmd.OutOrStdout()).Encode(res)
}
failed := 0
for _, r := range res {
if r.Err != "" {
failed++
fmt.Fprintf(cmd.OutOrStdout(), "%s failed: %s\n", r.URL, r.Err)
} else {
fmt.Fprintf(cmd.OutOrStdout(), "%s delivered\n", r.URL)
}
}
if failed > 0 {
return fmt.Errorf("%d of %d webhooks were not reached", failed, len(res))
}
return nil
},
})
return c
}
internal/cli/server.go
+42 -2
View File
@@ -31,6 +31,7 @@ import (
"golang.org/x/sync/errgroup"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
@@ -43,6 +44,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
@@ -118,6 +120,13 @@ func newServerCmd() *cobra.Command {
if err != nil {
return err
}
// The longest any key may live, where every way of making one
// reads it: the store.
if st := rt.Store(); st != nil {
if err := st.SetMaxKeyTTL(cmd.Context(), storepkg.ScopeGateway, "", cfg.Auth.MaxKeyTTL); err != nil {
log.Warn("could not record auth.max_key_ttl", "err", err)
}
}
// Guardrails, when server.yaml defines any. Attachments to
// parts of the tenancy tree live in the store; without one, only
// guard.default applies.
@@ -145,6 +154,24 @@ func newServerCmd() *cobra.Command {
rt.Start(ctx)
defer rt.Stop()
// Webhooks for what needs a person, when server.yaml names any.
notifier, err := notify.New(ctx, cfg.Notify, log)
if err != nil {
return err
}
rt.SetNotifier(notifier)
if gate != nil && notifier != nil {
gate.SetJudgeAlarm(func(model string, err error) {
notifier.Notify(notify.JudgeUnavailable, model,
fmt.Sprintf("the guardrails judge %s gives no verdict (%v): policies with fail: closed "+
"refuse what it should read, fail: open serve it unjudged", model, err),
map[string]any{"model": model, "error": err.Error()})
})
}
if notifier != nil {
log.Info("notifications on", "webhooks", len(cfg.Notify.Webhooks))
}
// The control plane replaces the static endpoint list: agents
// register, the reconciler places replicas, and Ready ones are
// published to the router (plan §6.4).
@@ -172,6 +199,7 @@ func newServerCmd() *cobra.Command {
return err
}
defer ctl.Close()
ctl.Reconciler().SetNotifier(notifier)
admin = ctl
agentPath, agentHandler = svc.Handler()
@@ -263,12 +291,14 @@ func newServerCmd() *cobra.Command {
}
adminAPI = api
api.SetReleases(cfg.Releases)
api.SetNotifier(notifier)
if st := rt.Store(); st != nil && gate != nil {
api.SetGuard(&adminapi.GuardAdmin{Library: cfg.Guard, Store: st, Invalidate: gate.Invalidate})
}
if st := rt.Store(); st != nil {
// Removing a node revokes its certificate.
api.SetRevoker(st.RevokeNode)
api.SetAudit(st)
}
// Catalog changes need somewhere to persist, so this is wired
// only when there is a store. Without one, the endpoints say so
@@ -374,7 +404,7 @@ func newServerCmd() *cobra.Command {
// changes it, and what replaces replicas.
Current: func() []catalog.Model { return admin.Catalog().Models() },
Models: catalogSvc, Restart: admin.Reconciler().Restart,
Pool: adminAPI.PoolNodes,
Pool: adminAPI.PoolNodes, Notify: notifier,
})
adminAPI.SetServe(serve)
updateMetrics.Set(serve)
@@ -404,7 +434,7 @@ func newServerCmd() *cobra.Command {
return err
}
adminSrv := withLimits(&http.Server{
Handler: router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, adminMux)),
Handler: router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, auditAPI(rt, adminMux, log))),
ReadTimeout: time.Minute,
})
@@ -660,6 +690,16 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
return ctl, svc, nil
}
// auditAPI records every change made through the admin API, when there is a
// store to record it in.
func auditAPI(rt *router.Router, next http.Handler, log *slog.Logger) http.Handler {
st := rt.Store()
if st == nil {
return next
}
return audit.API(st, adminapi.Actor, log, next)
}
// metricsAddr renders the metrics listener for the startup line, saying so
// plainly when there is none rather than printing an empty string.
func metricsAddr(ln net.Listener) string {
internal/cli/serverinit.go
+96 -7
View File
@@ -81,6 +81,9 @@ type serverInit struct {
// make it serve something: a mirror, this machine as a node, and a small
// model on it.
noMirror, noNode, noModel bool
// noGuardModel leaves out the guardrails' judge model; the pattern
// checks in the baseline policy stay.
noGuardModel bool
// noFirewall leaves this machine's firewall, and the store members',
// as they are.
noFirewall bool
@@ -164,6 +167,8 @@ func newServerInitCmd() *cobra.Command {
"use the mirror on another machine, set up there by 'llmgw mirror init': its management API, http://host:8082")
f.BoolVar(&in.noNode, "no-node", false, "do not make this machine a node (and serve no model on it)")
f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
f.BoolVar(&in.noGuardModel, "no-guard-model", false,
"do not fetch and serve the guardrails' judge model ("+guardModelName+"); the pattern checks stay")
f.BoolVar(&in.noFirewall, "no-firewall", false,
"leave the firewall alone, instead of keeping llmgw's internal ports to the cluster's networks")
f.BoolVar(&in.postgres, "postgres", false,
@@ -217,6 +222,21 @@ func (in *serverInit) run(ctx context.Context) error {
return fmt.Errorf("node: %w", err)
}
}
// The protections first: a starter model that fits nowhere must not
// leave the machine without its firewall or its judge.
if hasSystemd && !in.noFirewall {
if err := in.ensureFirewall(ctx); err != nil {
// The cluster works without it; said, not fatal.
fmt.Fprintf(in.out, " !! firewall: %v; llmgw's ports answer every network this machine is on\n", err)
}
}
if in.judgeWanted() && in.configJudgesWith(guardModelName) {
// Not fatal: without its judge the baseline policy still checks for
// secrets and injection, and a judged request is served flagged.
if err := in.ensureGuardModel(ctx); err != nil {
fmt.Fprintf(in.out, " !! guard model: %v; requests are not judged until it runs\n", err)
}
}
switch {
case !hasSystemd || in.noModel:
case in.noMirror && in.mirrorURL == "":
@@ -226,16 +246,75 @@ func (in *serverInit) run(ctx context.Context) error {
return fmt.Errorf("starter model: %w", err)
}
}
if hasSystemd && !in.noFirewall {
if err := in.ensureFirewall(ctx); err != nil {
// The cluster works without it; said, not fatal.
fmt.Fprintf(in.out, " !! firewall: %v; llmgw's ports answer every network this machine is on\n", err)
}
}
in.nextSteps()
return nil
}
// judgeWanted is whether this install fetches the judge model, and so
// whether the baseline policy asks it.
func (in *serverInit) judgeWanted() bool {
if in.noGuardModel || (in.noMirror && in.mirrorURL == "") || in.unitPath == "" {
return false
}
_, err := exec.LookPath("systemctl")
return err == nil
}
// configJudgesWith is whether server.yaml has a policy asking that judge: a
// configuration written before init installed one is never changed, so the
// model is not fetched for nothing.
func (in *serverInit) configJudgesWith(model string) bool {
cfg, err := config.Load(in.configPath())
if err != nil {
return false
}
for _, p := range cfg.Guard.Policies {
if p.Judge != nil && p.Judge.Model == model {
return true
}
}
return false
}
// guardSection is server.yaml's guard: the baseline every request passes,
// with the judge model when there is one.
func guardSection(judge bool) string {
j := ` # A judge model reads what patterns cannot: violence, weapons, abuse,
# self-harm. Installed by init; --no-guard-model leaves it out.
# judge: {model: ` + guardModelName + `, input: block, output: block, ...}
`
if judge {
j = ` # A judge model reads what patterns cannot: violence, weapons, abuse,
# self-harm. Two replicas run on CPUs with AVX2, about half a second
# for a short text; an answer is read every 600 characters as it
# streams. With fail: closed, a request the judge cannot read within
# latency_budget is refused (503); open serves it, flagged, unjudged.
# Defamation (S5), specialized advice (S6), intellectual property (S8)
# and elections (S13) are left out: they refuse too much ordinary work.
judge:
model: ` + guardModelName + `
input: block
output: block
categories: [S1, S2, S3, S4, S7, S9, S10, S11, S12]
latency_budget: 5s
`
}
return `# Guardrails (docs/guardrails.md). The baseline applies to every request;
# attach stricter policies to parts of the tree with 'llmgw ctl guard attach'.
# mode: shadow records what would be done without doing it.
guard:
default: [baseline]
policies:
baseline:
fail: closed
input:
pii: {secret: block, card: redact}
injection: flag
output:
pii: {secret: block, card: redact}
` + j
}
func (in *serverInit) done(format string, a ...any) {
fmt.Fprintf(in.out, " ok "+format+"\n", a...)
in.changed = true
@@ -448,7 +527,7 @@ func (in *serverInit) ensureConfig(context.Context) error {
contact = "platform@example.com"
}
body := fmt.Sprintf(serverConfigTemplate, in.etcDir, in.etcDir, in.etcDir,
in.storeAddress(), contact)
in.storeAddress(), contact, guardSection(in.judgeWanted()))
if err := os.WriteFile(path, []byte(body), 0o640); err != nil {
return err
}
@@ -503,10 +582,14 @@ auth:
default_price:
prompt_per_1k: 1000
completion_per_1k: 1000
# The longest any API key lives; a key made without an expiry gets this.
# Shorter per organization, team or project with 'llmgw keys ttl'.
max_key_ttl: 2160h # 90 days
landing:
contact: "%s"
%s
# The engines every node installs for itself. Omitted, each gets the llama.cpp
# build this release was checked against.
# engines:
@@ -700,5 +783,11 @@ The control plane is up. Next:
llmgw ctl model add -f model.yaml
The web UI is on port 8080; /admin once you are signed in as the administrator.
Before people outside this network use it:
# port 8080 is plain HTTP: put a proxy with a certificate in front
# (deploy/proxy/ in the release; Caddy gets the certificate itself)
# backups carry every secret: name who may open them, backup_recipients
# in server.yaml, and copy them off this machine
`)
}
internal/cli/serverinit_cluster.go
+105 -28
View File
@@ -41,6 +41,13 @@ const (
starterSource = "hf:Qwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf"
starterVersion = "starter"
// The guardrails' judge: Meta's Llama Guard 3 1B, which reads a prompt
// or an answer and says whether it is unsafe and why. Q4_K_M is about
// 900 MB and judges a short text in about half a second on a CPU.
guardModelName = "llama-guard-3-1b"
guardModelSource = "hf:QuantFactory/Llama-Guard-3-1B-GGUF/Llama-Guard-3-1B.Q4_K_M.gguf"
guardModelVersion = "guard"
// localLabel marks the control plane's own node, so the starter model is
// served where it was asked for and does not wander to the first GPU node
// that joins later.
@@ -436,46 +443,93 @@ func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model
}
}
// llamaGuard3Categories name Llama Guard 3's S1 to S13, so a verdict is
// recorded as "Violent Crimes" rather than "S1".
var llamaGuard3Categories = []string{"Violent Crimes", "Non-Violent Crimes", "Sex-Related Crimes",
"Child Sexual Exploitation", "Defamation", "Specialized Advice", "Privacy", "Intellectual Property",
"Indiscriminate Weapons", "Hate", "Suicide & Self-Harm", "Sexual Content", "Elections"}
// guardModel is the judge's catalog entry: CPU only, so it takes no card from
// the models people use, on whichever node has room.
func guardModel(source string) catalog.Model {
return catalog.Model{
Name: guardModelName,
Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
Source: source,
Role: catalog.RoleGuard,
Guard: &catalog.GuardModel{Format: "llamaguard3", Categories: llamaGuard3Categories},
// Two: every judged request waits on it, and with fail: closed a
// judge that is not there refuses everything.
Scaling: catalog.Scaling{Min: 2, Max: 2},
Profiles: []catalog.Profile{{
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2,
Args: []string{"--ctx-size", "8192"},
}},
}
}
// ensureStarter fetches the starter model into the mirror and serves it here.
func (in *serverInit) ensureStarter(ctx context.Context) error {
// Served here when this machine is a node, pinned so it does not wander
// to the first GPU node that joins; otherwise on whichever node joins.
pinned := !in.noNode
return in.ensureServed(ctx, starterName, starterSource, starterVersion, pinned,
func(url string, classes []string) catalog.Model { return starterModel(url, classes, pinned) })
}
// ensureGuardModel fetches the guardrails' judge model into the mirror and
// adds it to the catalog, to run on any node with room.
func (in *serverInit) ensureGuardModel(ctx context.Context) error {
return in.ensureServed(ctx, guardModelName, guardModelSource, guardModelVersion, false,
func(url string, _ []string) catalog.Model { return guardModel(url) })
}
// ensureServed fetches a model into the mirror, adds it to the catalog, and
// when it runs here, waits for it to answer.
func (in *serverInit) ensureServed(ctx context.Context, name, source, version string, here bool,
entry func(url string, gpuClasses []string) catalog.Model,
) error {
client := in.adminAPI()
var models []adminapi.ModelView
if err := client.get(ctx, "/api/v1/models", &models); err != nil {
return fmt.Errorf("reading the catalog: %w", err)
}
for _, m := range models {
if m.Name == starterName {
in.kept("%s in the catalog", starterName)
return in.waitForStarter(ctx)
if m.Name == name {
in.kept("%s in the catalog", name)
if !here {
return nil
}
return in.waitServing(ctx, name)
}
}
upstream, ok := sourcecheck.HTTPSource(starterSource)
upstream, ok := sourcecheck.HTTPSource(source)
if !ok {
return fmt.Errorf("cannot turn %s into a download", starterSource)
return fmt.Errorf("cannot turn %s into a download", source)
}
m, err := mirror.NewClient(in.mirrorAPIURL(), tlsconf.Config{}, 15*time.Second)
if err != nil {
return err
}
st, err := m.Status(ctx, starterName, starterVersion)
st, err := m.Status(ctx, name, version)
var apiErr *mirror.APIError
switch {
case err == nil:
case errors.As(err, &apiErr) && apiErr.Status == 404:
if st, err = m.Fetch(ctx, mirror.FetchRequest{
Source: upstream, Model: starterName, Version: starterVersion,
Source: upstream, Model: name, Version: version,
}); err != nil {
return fmt.Errorf("asking the mirror for %s: %w", starterName, err)
return fmt.Errorf("asking the mirror for %s: %w", name, err)
}
fmt.Fprintf(in.out, " .. downloading %s into the mirror\n", starterName)
fmt.Fprintf(in.out, " .. downloading %s into the mirror\n", name)
default:
return err
}
lastPct := -10
err = waitFor(ctx, 30*time.Minute, starterName+" to download", func() (bool, error) {
cur, err := m.Status(ctx, starterName, starterVersion)
err = waitFor(ctx, 30*time.Minute, name+" to download", func() (bool, error) {
cur, err := m.Status(ctx, name, version)
if err != nil {
return false, nil
}
@@ -484,7 +538,7 @@ func (in *serverInit) ensureStarter(ctx context.Context) error {
case mirror.Present:
return true, nil
case mirror.Failed:
return false, fmt.Errorf("the mirror could not download %s: %s", starterName, cur.Status.Err)
return false, fmt.Errorf("the mirror could not download %s: %s", name, cur.Status.Err)
}
if pct, ok := cur.Status.Percent(); ok && pct >= lastPct+10 {
fmt.Fprintf(in.out, " .. %d%%\n", pct)
@@ -495,45 +549,68 @@ func (in *serverInit) ensureStarter(ctx context.Context) error {
if err != nil {
return err
}
in.done("%s in the mirror", starterName)
in.done("%s in the mirror", name)
// Served here when this machine is a node, pinned so it does not wander
// to the first GPU node that joins; otherwise on whichever node joins.
pinned := !in.noNode
var classes []string
if pinned {
if here {
classes, _ = gpuClasses(ctx)
}
body, err := json.Marshal(starterModel(st.URL, classes, pinned))
body, err := json.Marshal(entry(st.URL, classes))
if err != nil {
return err
}
if err := client.post(ctx, "/api/v1/models", body, &map[string]string{}); err != nil {
return fmt.Errorf("adding %s to the catalog: %w", starterName, err)
return fmt.Errorf("adding %s to the catalog: %w", name, err)
}
if !pinned {
in.done("%s in the catalog; it starts on the first node that joins", starterName)
if !here {
in.done("%s in the catalog; it runs on a node with room for it", name)
return nil
}
where := "on CPU"
if len(classes) > 0 {
where = "on " + strings.Join(classes, ", ") + ", or CPU if the card is busy"
}
in.done("%s in the catalog, served here %s", starterName, where)
return in.waitForStarter(ctx)
in.done("%s in the catalog, served here %s", name, where)
return in.waitServing(ctx, name)
}
func (in *serverInit) waitForStarter(ctx context.Context) error {
// unplacedGrace is how long init lets placement say a model fits nowhere
// before believing it: long enough for a node still installing its engine.
var unplacedGrace = 90 * time.Second
func (in *serverInit) waitServing(ctx context.Context, name string) error {
client := in.adminAPI()
last := ""
return waitFor(ctx, 10*time.Minute, starterName+" to start", func() (bool, error) {
var unplaced time.Time
return waitFor(ctx, 10*time.Minute, name+" to start", func() (bool, error) {
var reps []adminapi.ReplicaView
if err := client.get(ctx, "/api/v1/replicas?model="+starterName, &reps); err != nil {
if err := client.get(ctx, "/api/v1/replicas?model="+name, &reps); err != nil {
return false, nil
}
// Nowhere to put it is not waited out for ten minutes: once
// placement has said why for unplacedGrace, that is the answer.
if len(reps) == 0 {
var ex adminapi.ExplainView
if err := client.get(ctx, "/api/v1/explain-placement?model="+name, &ex); err == nil &&
!ex.Placed && len(ex.Rejections) > 0 {
if unplaced.IsZero() {
unplaced = time.Now()
}
if time.Since(unplaced) > unplacedGrace {
var why []string
for _, r := range ex.Rejections {
why = append(why, r.Node+": "+r.Reason)
}
return false, fmt.Errorf("%s has nowhere to run (%s); add a node with room, "+
"or free some ('llmgw ctl explain-placement %s')", name, strings.Join(why, "; "), name)
}
return false, nil
}
}
unplaced = time.Time{}
for _, r := range reps {
if r.State == "ready" {
in.kept("%s is serving (profile %s)", starterName, r.Profile)
in.kept("%s is serving (profile %s)", name, r.Profile)
return true, nil
}
}
@@ -543,7 +620,7 @@ func (in *serverInit) waitForStarter(ctx context.Context) error {
line += ": " + r.Err
}
if line != last {
fmt.Fprintf(in.out, " .. %s %s\n", starterName, line)
fmt.Fprintf(in.out, " .. %s %s\n", name, line)
last = line
}
}
internal/cli/serverinit_cluster_test.go
+86
View File
@@ -1,12 +1,15 @@
package cli
import (
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
@@ -121,3 +124,86 @@ func TestTheMirrorAPIOnlyAnswersItsControlPlane(t *testing.T) {
t.Error("nonsense was accepted")
}
}
func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
for _, judge := range []bool{false, true} {
body := fmt.Sprintf(serverConfigTemplate, "/etc/llmgw", "/etc/llmgw", "/etc/llmgw",
"/var/lib/llmgw/llmgw.db", "ops@example.com", guardSection(judge))
path := filepath.Join(t.TempDir(), "server.yaml")
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("judge=%v: %v", judge, err)
}
if err := cfg.Validate(); err != nil {
t.Fatalf("judge=%v: %v", judge, err)
}
if len(cfg.Guard.Default) != 1 || cfg.Guard.Default[0] != "baseline" {
t.Errorf("judge=%v: guard.default %v", judge, cfg.Guard.Default)
}
p := cfg.Guard.Policies["baseline"]
if p.Fail != "closed" {
t.Errorf("judge=%v: the baseline fails %q", judge, p.Fail)
}
if (p.Judge != nil) != judge {
t.Errorf("judge=%v: the policy's judge is %+v", judge, p.Judge)
}
if judge && (p.Judge.Model != guardModelName || p.Judge.Input != "block") {
t.Errorf("judge %+v", p.Judge)
}
if cfg.Auth.MaxKeyTTL != 90*24*time.Hour || cfg.Auth.SignupOpen || cfg.DebugPprof || cfg.AdminRemote {
t.Errorf("not hardened: ttl %v signup %v pprof %v remote %v",
cfg.Auth.MaxKeyTTL, cfg.Auth.SignupOpen, cfg.DebugPprof, cfg.AdminRemote)
}
}
}
func TestTheGuardModelIsAValidJudge(t *testing.T) {
m := guardModel("http://10.0.0.5:8081/llama-guard-3-1b/guard/g.gguf")
c, err := catalog.New([]catalog.Model{m})
if err != nil {
t.Fatalf("not a valid catalog entry: %v", err)
}
got, _ := c.Resolve(guardModelName)
if !got.IsGuard() || got.Guard.Format != "llamaguard3" || len(got.Guard.Categories) != 13 || got.Scaling.Min != 2 {
t.Errorf("%+v", got)
}
for _, p := range got.Profiles {
if p.GPUClass != "" {
t.Errorf("the judge takes a card: %+v", p)
}
}
}
func TestInitSaysAtOnceWhenAModelFitsNowhere(t *testing.T) {
api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/api/v1/replicas":
w.Write([]byte(`[]`))
case "/api/v1/explain-placement":
w.Write([]byte(`{"model":"qwen","rejections":[{"profile":"cpu","node":"n1","reason":"has 3 GB of memory with 1 free, needs 2"}]}`))
default:
http.NotFound(w, r)
}
}))
defer api.Close()
dir := t.TempDir()
token := filepath.Join(dir, "token")
os.WriteFile(token, []byte(strings.Repeat("t", 40)), 0o600)
os.WriteFile(filepath.Join(dir, "server.yaml"),
[]byte("admin_listen: "+strings.TrimPrefix(api.URL, "http://")+"\nadmin_token_file: "+token+"\n"), 0o600)
defer func(g time.Duration) { unplacedGrace = g }(unplacedGrace)
unplacedGrace = 0
in := &serverInit{etcDir: dir, out: io.Discard}
start := time.Now()
err := in.waitServing(t.Context(), "qwen")
if err == nil || !strings.Contains(err.Error(), "needs 2") || !strings.Contains(err.Error(), "explain-placement") {
t.Fatalf("%v", err)
}
if time.Since(start) > 10*time.Second {
t.Errorf("took %s to say so", time.Since(start))
}
}
internal/cli/tui_actions_test.go
+20
View File
@@ -648,4 +648,24 @@ func TestGuardrailsSayWhereTheyApplyFrom(t *testing.T) {
if items := m.orgMenu(m.people.orgs[0]).items; !slices.ContainsFunc(items, func(i menuItem) bool { return i.label == "Guardrails…" }) {
t.Error("no Guardrails… in the menu")
}
if items := m.orgMenu(m.people.orgs[0]).items; !slices.ContainsFunc(items, func(i menuItem) bool { return i.label == "Longest key lifetime…" }) {
t.Error("no Longest key lifetime… in the menu")
}
if items := m.keyMenu(store.KeyRow{}).items; !slices.ContainsFunc(items, func(i menuItem) bool { return i.label == "Rotate…" }) {
t.Error("no Rotate… on a key")
}
}
func TestTheAuditViewListsChangesAndOpensOne(t *testing.T) {
m, _ := consoleWith(t, cAudit)
m.width, m.height = 160, 40
m.data.audit = []store.AuditEvent{{At: time.Now(), Actor: "ilker", Source: "api",
Action: "POST /api/v1/models", Target: "qwen", Status: 200, Detail: `{"name":"qwen"}`}}
if v := m.View(); !strings.Contains(v, "POST /api/v1/models") || !strings.Contains(v, "ilker") {
t.Fatalf("not listed:\n%s", v)
}
tm := press(m, "enter")
if v := tm.View(); !strings.Contains(v, `"name":"qwen"`) {
t.Errorf("not opened:\n%s", v)
}
}
internal/cli/tui_audit.go
+71
View File
@@ -0,0 +1,71 @@
package cli
import (
"fmt"
"strconv"
"strings"
"time"
tea "github.com/charmbracelet/bubbletea"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// The audit view: who changed what, newest first — the same as `llmgw ctl
// audit`.
func (m consoleModel) auditPanel(pw int) (string, string, string) {
keys := "→ or Enter open · ↑↓ choose · Q leave"
if m.inPanel {
keys = "↑↓ row · Enter what it was given · ← back · Q leave"
}
head := "\n " + sDim.Render("Every change through ctl, this console and the /admin pages, the last week.") + "\n\n"
var rows [][]string
for _, e := range m.data.audit {
status := strconv.Itoa(e.Status)
if e.Status >= 400 {
status = sBad.Render(status)
}
rows = append(rows, []string{ageShort(e.At), e.Actor, e.Source, e.Action, dash(e.Target), status})
}
if len(rows) == 0 {
return "Audit", head + " " + sDim.Render("Nothing changed in the last week, or this control plane has no store."), keys
}
return "Audit", head + table([]string{"WHEN", "WHO", "VIA", "WHAT", "TARGET", "STATUS"}, rows, m.row, pw, m.inPanel), keys
}
// auditDialog is one change in full.
func auditDialog(e store.AuditEvent) (string, string) {
var b strings.Builder
fmt.Fprintf(&b, "%s, by %s through %s\nstatus %d\n", e.At.Format(time.DateTime), e.Actor, e.Source, e.Status)
if e.Target != "" {
fmt.Fprintf(&b, "target %s\n", e.Target)
}
if e.Detail != "" && e.Detail != "{}" {
fmt.Fprintf(&b, "\n%s\n", wrap(e.Detail, 80, ""))
}
b.WriteString("\n" + sDim.Render("any key"))
return e.Action, b.String()
}
// usageExportForm writes usage to a CSV file on this machine, as `llmgw ctl
// usage export` does.
func (m *consoleModel) usageExportForm() {
now := time.Now().UTC()
first := time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.UTC)
m.openForm(&formDialog{title: "Export usage as CSV",
fields: []formField{
textF("from", first.AddDate(0, -1, 0).Format("2006-01-02"), "first day"),
textF("to", first.Format("2006-01-02"), "the day after the last"),
textF("period", "day", "hour, day or month"),
textF("by", "project,model", "org, team, project, user, key, model"),
textF("file", "/tmp/llmgw-usage-"+first.AddDate(0, -1, 0).Format("2006-01")+".csv", "written on this machine"),
},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
if v[4] == "" {
return nil, fmt.Errorf("a file to write")
}
return m.runCLI("Export usage", "ctl", "usage", "export", "--from", v[0], "--to", v[1],
"--period", v[2], "--by", v[3], "--output", v[4]), nil
}})
}
internal/cli/tui_console.go
+22 -2
View File
@@ -24,6 +24,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// The console: what a bare `sudo llmgw` opens on a control plane. Down the
@@ -46,6 +47,7 @@ const (
cAccounts
cKeys
cOrgs
cAudit
cManual
cSettings
cAddNode
@@ -58,7 +60,7 @@ const (
cCount
)
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Manual",
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
"server.yaml", "Add a node", "Replicate the store", "Serve a model", "Upgrade the cluster",
"Restart all replicas", "Back up now", "Leave"}
@@ -75,6 +77,7 @@ type consoleData struct {
models []adminapi.ModelView
replicas []adminapi.ReplicaView
jobs []modelserve.Job
audit []store.AuditEvent
err error
}
@@ -228,6 +231,7 @@ func (m consoleModel) fetch() tea.Cmd {
_ = c.get(ctx, "/api/v1/models", &d.models)
_ = c.get(ctx, "/api/v1/replicas", &d.replicas)
_ = c.get(ctx, "/api/v1/jobs", &d.jobs)
_ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit)
return consoleDataMsg(d)
}
}
@@ -474,6 +478,9 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
case cJobs:
m.dialog = "error"
m.dialogTitle, m.dialogText = jobDialog(m.data.jobs[m.row])
case cAudit:
m.dialog = "error"
m.dialogTitle, m.dialogText = auditDialog(m.data.audit[m.row])
case cServe:
return m, m.fetchRepo(m.hf.models[m.row].ID)
case cAccounts:
@@ -529,6 +536,15 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
case cOrgs:
m.createOrgForm()
}
case "n":
if m.menu == cMonitor {
return m, m.runCLI("Test the webhooks", "ctl", "--server="+m.client.base, "notify", "test")
}
case "e":
if m.menu == cUsage {
m.usageExportForm()
return m, nil
}
case "right", "l":
if m.menu == cUsage {
m.usageBy = (m.usageBy + 1) % len(usageGroupings)
@@ -873,6 +889,8 @@ func (m consoleModel) rowCount() int {
return len(m.mirrorGroups())
case cJobs:
return len(m.data.jobs)
case cAudit:
return len(m.data.audit)
case cServe:
return len(m.hf.models)
case cAccounts:
@@ -1083,7 +1101,7 @@ func (m consoleModel) panel() (title, body, keys string) {
compactNum(u.Errors), fmt.Sprintf("%.2f", float64(u.CostMicros)/1e6)})
}
if m.inPanel {
keys = "↑↓ row · → grouping · ← back · Q leave"
keys = "↑↓ row · → grouping · E export as CSV · ← back · Q leave"
}
return "Usage", head + table([]string{"WHO", "REQUESTS", "TOKENS", "ERRORS", "SPENT"}, rows, m.row, pw, m.inPanel), keys
case cQuotas:
@@ -1108,6 +1126,8 @@ func (m consoleModel) panel() (title, body, keys string) {
return m.mirrorPanel(pw)
case cJobs:
return m.jobsPanel(pw)
case cAudit:
return m.auditPanel(pw)
case cMonitor:
return m.monitorPanel(pw)
case cServe:
internal/cli/tui_docs.go
+2 -2
View File
@@ -25,8 +25,8 @@ type docTopic struct {
}
// docOrder is the documents in the order someone new reads them.
var docOrder = []string{"console", "quickstart", "quick-install", "install", "node-onboarding", "onboarding",
"choosing-models", "fabrics", "facility-aware-placement", "data", "reference"}
var docOrder = []string{"console", "quick-install", "install", "node-onboarding", "onboarding",
"guardrails", "choosing-models", "fabrics", "facility-aware-placement", "data", "reference", "development"}
func docTopics() []docTopic {
read := func(path string) func() string {
internal/cli/tui_init.go
+5 -1
View File
@@ -54,6 +54,7 @@ const (
fMirror
fNode
fModel
fGuard
fStart
fCount
)
@@ -104,6 +105,7 @@ func newInitModel(in *serverInit, found []string, askForm bool) initModel {
m.toggles[fMirror] = !in.noMirror
m.toggles[fNode] = !in.noNode
m.toggles[fModel] = !in.noModel
m.toggles[fGuard] = !in.noGuardModel
if askForm {
m.admin.Focus()
}
@@ -216,6 +218,7 @@ func (m initModel) formKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
m.in.noMirror = !m.toggles[fMirror]
m.in.noNode = !m.toggles[fNode]
m.in.noModel = !m.toggles[fModel]
m.in.noGuardModel = !m.toggles[fGuard]
m.form = false
return m, tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
}
@@ -228,7 +231,7 @@ func (m initModel) formKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
var cmd tea.Cmd
m.hosts, cmd = m.hosts.Update(k)
return m, cmd
case fPostgres, fMirror, fNode, fModel:
case fPostgres, fMirror, fNode, fModel, fGuard:
if k.String() == " " || k.String() == "x" {
m.toggles[m.focus] = !m.toggles[m.focus]
}
@@ -371,6 +374,7 @@ func (m initModel) formView() string {
}
b.WriteString(check(fNode, "This machine as a node", "serves models here too, on its GPUs or CPU"))
b.WriteString(check(fModel, "Starter model", "a small Qwen, so there is something to ask at once"))
b.WriteString(check(fGuard, "Guardrails' judge", "Llama Guard 3 1B on a CPU, refusing violent, criminal and abusive requests"))
b.WriteString("\n" + label(fStart, lipgloss.NewStyle().Bold(true).Render("Set up")) + "\n")
if m.formError != "" {
b.WriteString("\n" + sBad.Render(m.formError) + "\n")
internal/cli/tui_init_test.go
+2 -1
View File
@@ -23,12 +23,13 @@ func TestTheInitFormBecomesInitsSettings(t *testing.T) {
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // PostgreSQL on
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab},
tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // starter model off
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // judge off
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyEnter})
if in.admin != "ops@example.com" || strings.Join(in.hosts, "|") != "llm.example.com|203.0.113.10" {
t.Errorf("admin %q hosts %q", in.admin, in.hosts)
}
if !in.postgres || in.noMirror || in.noNode || !in.noModel {
if !in.postgres || in.noMirror || in.noNode || !in.noModel || !in.noGuardModel {
t.Errorf("toggles: postgres %v mirror %v node %v model %v", in.postgres, !in.noMirror, !in.noNode, !in.noModel)
}
if m.(initModel).form {
internal/cli/tui_monitor.go
+1 -1
View File
@@ -258,7 +258,7 @@ func side(w int, a, b string) string {
}
func (m consoleModel) monitorPanel(pw int) (string, string, string) {
keys := "→ or Enter open · ↑↓ choose · Q leave"
keys := "→ or Enter open · ↑↓ choose · N test the webhooks · Q leave"
if m.inPanel {
keys = "1–4 or R range · PgUp/PgDn scroll · ← back · Q leave"
}
internal/cli/tui_people.go
+54 -10
View File
@@ -204,11 +204,10 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
return nil
}})
}
if u.IsAdmin {
items = append(items, cli("Take away the cluster dashboard", "users", "demote", u.ID))
} else {
items = append(items, cli("Give the cluster dashboard (/admin)", "users", "promote", u.ID))
}
items = append(items, menuItem{"Staff role (" + or(u.StaffRole(), "none") + ")…", func(m *consoleModel) tea.Cmd {
m.menuD = staffRoleMenu(u)
return nil
}})
items = append(items,
menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Membership for " + who,
@@ -264,6 +263,28 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
return &menuDialog{title: who, items: items}
}
// staffRoleMenu sets what a person may do on the pages under /admin.
func staffRoleMenu(u store.User) *menuDialog {
about := map[string]string{
store.RoleViewer: "looks at every administration page, changes nothing",
store.RoleOperator: "also runs the cluster: models, the mirror, nodes",
store.RoleBilling: "sees usage and quotas",
store.RoleAdmin: "may do anything",
"none": "no part in running the gateway",
}
var items []menuItem
for _, role := range append(slices.Clone(store.StaffRoles), "none") {
label := role + " — " + about[role]
if role == or(u.StaffRole(), "none") {
label = "✓ " + label
}
items = append(items, menuItem{label, func(m *consoleModel) tea.Cmd {
return m.runCLI("Staff role for "+u.Email, "users", "role", u.ID, role)
}})
}
return &menuDialog{title: "Staff role · " + u.Email, items: items}
}
func (m *consoleModel) createUserForm() {
m.openForm(&formDialog{title: "New account",
note: "Created already approved. The password should be changed by its owner.",
@@ -290,6 +311,18 @@ func (m consoleModel) keyMenu(k store.KeyRow) *menuDialog {
name := k.Name + " (" + k.Prefix + ")"
return &menuDialog{title: name, items: []menuItem{
{"Its limits…", func(m *consoleModel) tea.Cmd { m.limitsForm(store.ScopeKey, k.ID, name); return nil }},
{"Rotate…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Rotate " + name,
note: "A new key with the same project, owner and limits is shown once; this one keeps working for the overlap, then stops.",
fields: []formField{textF("overlap", "24h", "how long both work, like 48h; 0 stops this one now")},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
if _, err := time.ParseDuration(v[0]); err != nil {
return nil, fmt.Errorf("overlap is a duration, like 24h")
}
return m.runCLI("Rotate "+name, "keys", "rotate", k.ID, "--overlap="+v[0]), nil
}})
return nil
}},
{"Revoke…", func(m *consoleModel) tea.Cmd {
m.confirmD = &confirmDialog{title: "Revoke " + name,
text: "It stops working at once. It is kept, revoked, for the usage that refers to it.",
@@ -420,6 +453,20 @@ func (m consoleModel) orgMenu(r orgRow) *menuDialog {
return nil
}},
menuItem{"Guardrails…", func(m *consoleModel) tea.Cmd { m.menuD = m.guardMenu(r); return nil }},
menuItem{"Longest key lifetime…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Longest key lifetime · " + r.path,
note: "Keys made here get at most this, and this when none is asked for. Existing keys are not changed.",
fields: []formField{textF("longest", "", "like 720h; off removes the limit here")},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
if v[0] != "off" {
if d, err := time.ParseDuration(v[0]); err != nil || d <= 0 {
return nil, fmt.Errorf("a duration like 720h, or off")
}
}
return m.runCLI("Key lifetime "+r.path, "keys", "ttl", r.level+":"+r.id, v[0]), nil
}})
return nil
}},
menuItem{"Capacity quota…", func(m *consoleModel) tea.Cmd {
q := adminapi.QuotaView{Scope: r.level + ":" + r.id}
for _, x := range m.quotas {
@@ -565,16 +612,13 @@ func (m consoleModel) peoplePanel(pw int) (string, string, string) {
var rows [][]string
switch m.menu {
case cAccounts:
head = []string{"EMAIL", "STATE", "ADMIN", "MEMBER OF", "SIGNED UP", "LAST LOGIN"}
head = []string{"EMAIL", "STATE", "STAFF", "MEMBER OF", "SIGNED UP", "LAST LOGIN"}
for _, u := range m.people.users {
state := string(u.State)
if u.State == store.UserPending {
state = sWarn.Render(state)
}
admin := ""
if u.IsAdmin {
admin = "yes"
}
admin := u.StaffRole()
where := strings.Join(m.people.where[u.ID], "; ")
if where == "" {
where = sDim.Render("on their own")
internal/cli/usage.go
+52
View File
@@ -3,12 +3,15 @@ package cli
import (
"encoding/json"
"fmt"
"net/url"
"os"
"text/tabwriter"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// newUsageCmd shows what each group of the tenancy tree spent.
@@ -17,6 +20,54 @@ import (
// anything could show what any of them had used: usage came back per key, per
// user and per model only. A quota whose consumption cannot be seen is one
// nobody reviews until the day it refuses somebody.
func newUsageExportCmd(client func() *ctlClient) *cobra.Command {
var from, to, period, by, output string
c := &cobra.Command{
Use: "export",
Short: "Usage as CSV, by period and by organization, team, project, user, key or model",
Long: "Totals from the hourly and daily sums, so it reaches back past the request\n" +
"records' retention. Periods are UTC. Cost is given in micro-units and in\n" +
"units; names, not ids, except for keys.",
Example: " llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model > september.csv\n" +
" llmgw ctl usage export --period month --by org --output 2026.csv",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
q := url.Values{"period": {period}, "by": {by}, "format": {"json"}}
if from != "" {
q.Set("from", from)
}
if to != "" {
q.Set("to", to)
}
var rows []store.ExportRow
if err := client().get(cmd.Context(), "/api/v1/usage/export?"+q.Encode(), &rows); err != nil {
return err
}
out := cmd.OutOrStdout()
if output != "" {
f, err := os.Create(output)
if err != nil {
return err
}
defer f.Close()
out = f
}
adminapi.WriteExportCSV(out, rows)
if output != "" {
cmd.PrintErrf("%d rows written to %s\n", len(rows), output)
}
return nil
},
}
f := c.Flags()
f.StringVar(&from, "from", "", "first day, 2026-09-01 (default: the first of this month)")
f.StringVar(&to, "to", "", "the day after the last, 2026-10-01 (default: now)")
f.StringVar(&period, "period", "day", "hour, day or month")
f.StringVar(&by, "by", "project,model", "any of org, team, project, user, key, model, comma-separated")
f.StringVarP(&output, "output", "o", "", "write to this file instead of standard output")
return c
}
func newUsageCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
var by string
var since time.Duration
@@ -70,5 +121,6 @@ func newUsageCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
}
c.Flags().StringVar(&by, "by", "team", "level to group by: org, team or project")
c.Flags().DurationVar(&since, "since", 24*time.Hour, "how far back to total")
c.AddCommand(newUsageExportCmd(client))
return c
}
internal/cli/users.go
+38 -1
View File
@@ -37,7 +37,8 @@ func newUsersCmd() *cobra.Command {
newUserCreateCmd(open), newUserResetPasswordCmd(open),
newUserJoinCmd(open), newUserLeaveCmd(open), newUserDeleteCmd(open),
newUserAdminCmd(open, "promote", true, "Let an account see the cluster dashboard at /admin"),
newUserAdminCmd(open, "demote", false, "Take away access to the cluster dashboard"))
newUserAdminCmd(open, "demote", false, "Take away access to the cluster dashboard"),
newUserRoleCmd(open))
return cmd
}
@@ -177,6 +178,42 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
}
}
func newUserRoleCmd(open userOpener) *cobra.Command {
return &cobra.Command{
Use: "role <user> <viewer|operator|billing|admin|none>",
Short: "Give an account a part in running the gateway, from the pages under /admin",
Long: "viewer looks at every administration page and changes nothing. operator\n" +
"also runs the cluster from them: models through the mirror, a node's\n" +
"engines. billing sees usage and quotas. admin may do anything. none takes\n" +
"the role away. Every change made from those pages is in `llmgw ctl audit`.",
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
role := args[1]
if role == "none" {
role = ""
}
s, err := open(cmd)
if err != nil {
return err
}
defer s.Close()
u, err := resolveUser(cmd.Context(), s, args[0])
if err != nil {
return err
}
if err := s.SetStaffRole(cmd.Context(), u.ID, role); err != nil {
return err
}
if role == "" {
fmt.Fprintf(cmd.OutOrStdout(), "%s has no part in running the gateway now\n", u.Email)
} else {
fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, role)
}
return nil
},
}
}
func newUserAdminCmd(open userOpener, verb string, admin bool, short string) *cobra.Command {
return &cobra.Command{
Use: verb + " <user>",
internal/config/config.go
+22
View File
@@ -19,6 +19,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
@@ -128,6 +129,11 @@ type Server struct {
// (llmgw ctl guard attach). Empty checks nothing.
Guard guard.Config `koanf:"guard"`
// Notify sends what needs a person — a model parked, a node lost, a
// budget spent, new weights published — to webhooks, signed, without a
// Prometheus and Alertmanager in between. Empty sends nothing.
Notify notify.Config `koanf:"notify"`
// FirewallNetworks are networks, besides those each machine reaches the
// cluster on, that llmgw's internal ports answer: a monitoring host on
// another subnet, nodes behind a router. Written into every machine's
@@ -290,6 +296,16 @@ type Auth struct {
// means accounts are made with `llmgw users create` instead.
SignupOpen bool `koanf:"signup_open"`
// OIDC is sign-in through an identity provider.
OIDC OIDC `koanf:"oidc"`
// SCIM is account provisioning by the identity provider.
SCIM SCIM `koanf:"scim"`
// MaxKeyTTL is how long any API key may live; a key asked for with no
// expiry is given this. Organizations, teams and projects can set
// shorter ones (llmgw keys ttl). Zero is no limit.
MaxKeyTTL time.Duration `koanf:"max_key_ttl"`
// Personal is what a person may do on their own, outside any project: make
// personal keys and use the chat page personally, within these limits.
Personal Personal `koanf:"personal"`
@@ -646,9 +662,15 @@ func (s Server) Validate() error {
if _, err := s.FirewallAllow(); err != nil {
return err
}
if err := s.Notify.Validate(); err != nil {
return err
}
if err := s.Guard.Validate(); err != nil {
return err
}
if err := s.Auth.OIDC.Validate(); err != nil {
return err
}
if err := s.AgentTLS.Validate(); err != nil {
return err
}
internal/config/engines_test.go
+18
View File
@@ -36,3 +36,21 @@ func TestEachNodeGetsTheEnginesItsOverridesSay(t *testing.T) {
}
}
}
func TestOIDCSettingsAreCheckedBeforeAProviderIsAsked(t *testing.T) {
ok := OIDC{Issuer: "https://login.example.com/realms/acme", ClientID: "llmgw",
Roles: map[string][]string{"admin": {"g"}}, Memberships: map[string]string{"ml": "team:Acme/ml"}}
if err := ok.Validate(); err != nil {
t.Fatal(err)
}
for name, o := range map[string]OIDC{
"plain http": {Issuer: "http://login.example.com", ClientID: "x"},
"no client": {Issuer: "https://login.example.com"},
"unknown role": {Issuer: "https://l", ClientID: "x", Roles: map[string][]string{"god": {"g"}}},
"bad place": {Issuer: "https://l", ClientID: "x", Memberships: map[string]string{"g": "galaxy:x"}},
} {
if o.Validate() == nil {
t.Errorf("%s accepted", name)
}
}
}
internal/config/oidc.go
+97
View File
@@ -0,0 +1,97 @@
package config
import (
"errors"
"fmt"
"net/url"
"strings"
)
// OIDC is sign-in through an identity provider: Keycloak, Entra ID, Google,
// Okta, Authentik — anything that speaks OpenID Connect.
//
// auth:
// oidc:
// issuer: https://login.example.com/realms/acme
// client_id: llmgw
// client_secret_file: /etc/llmgw/secrets/oidc-client-secret
// label: Acme SSO
// create_accounts: true
// roles: {admin: [llmgw-admins], operator: [llmgw-ops]}
// memberships: {ml-team: "team:Acme/ml"}
type OIDC struct {
// Issuer is the provider's address; its /.well-known/openid-configuration
// says the rest.
Issuer string `koanf:"issuer"`
// ClientID and the file holding its secret, as registered with the
// provider.
ClientID string `koanf:"client_id"`
ClientSecretFile string `koanf:"client_secret_file"`
// RedirectURL is where the provider sends people back:
// https://<gateway>/login/oidc/callback. Empty works it out from the
// request, which is right unless a proxy changes the address.
RedirectURL string `koanf:"redirect_url"`
// Label is what the sign-in button says: "Sign in with <label>".
Label string `koanf:"label"`
// Scopes asked for besides openid; empty is email and profile, and
// groups when Roles or Memberships use them.
Scopes []string `koanf:"scopes"`
// GroupsClaim names the ID token claim holding groups; empty is
// "groups".
GroupsClaim string `koanf:"groups_claim"`
// CreateAccounts makes an account, active, for a person the provider
// vouches for and the gateway does not know. Off, only existing accounts
// sign in this way.
CreateAccounts bool `koanf:"create_accounts"`
// Roles maps a staff role (viewer, operator, billing, admin) to the
// groups that give it. Set at every sign-in: someone taken out of the
// group loses the role the next time.
Roles map[string][]string `koanf:"roles"`
// Memberships maps a group to a place in the tree — org:NAME,
// team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in.
Memberships map[string]string `koanf:"memberships"`
// PasswordSignIn off refuses passwords on the sign-in form: people come
// in through the provider only. Administrators still can, so a broken
// provider does not lock everybody out.
NoPasswords bool `koanf:"no_passwords"`
}
// Enabled reports whether OIDC sign-in is configured.
func (o OIDC) Enabled() bool { return o.Issuer != "" }
// Validate checks the settings before a provider is asked anything.
func (o OIDC) Validate() error {
if !o.Enabled() {
return nil
}
u, err := url.Parse(o.Issuer)
if err != nil || u.Scheme != "https" && !(u.Scheme == "http" && (u.Hostname() == "localhost" || u.Hostname() == "127.0.0.1")) {
return fmt.Errorf("auth.oidc.issuer %q: an https address", o.Issuer)
}
if o.ClientID == "" {
return errors.New("auth.oidc.client_id is required")
}
for role := range o.Roles {
switch role {
case "viewer", "operator", "billing", "admin":
default:
return fmt.Errorf("auth.oidc.roles: %q is not viewer, operator, billing or admin", role)
}
}
for g, where := range o.Memberships {
kind, path, ok := strings.Cut(where, ":")
if !ok || path == "" || (kind != "org" && kind != "team" && kind != "project") {
return fmt.Errorf("auth.oidc.memberships.%s is %q: org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", g, where)
}
}
return nil
}
// SCIM lets an identity provider provision accounts (/scim/v2): made,
// suspended and reinstated as they are there, with groups mapped by
// auth.oidc's roles and memberships.
type SCIM struct {
// TokenFile holds the bearer token the provider sends. Empty turns
// provisioning off.
TokenFile string `koanf:"token_file"`
}
internal/control/adminapi/api.go
+12
View File
@@ -35,11 +35,14 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// API serves the management endpoints.
type API struct {
// notifier is the webhooks `ctl notify test` reaches; nil without any.
notifier *notify.Notifier
// releases is where release binaries are fetched from, for upgrades.
releases string
// storeStatus reads a replicated store's members; nil for one database.
@@ -48,6 +51,8 @@ type API struct {
serve *modelserve.Server
// history is the Prometheus the console's charts read from.
history History
// audit reads the audit log.
audit AuditReader
// guard is the guardrail policies' library and attachments.
guard *GuardAdmin
// revoke records a removed node, so its certificate is refused.
@@ -80,6 +85,9 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
a.registerQuotas(mux)
a.registerGuard(mux)
mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)
mux.HandleFunc("POST /api/v1/notify/test", a.notifyTest)
mux.HandleFunc("POST /api/v1/models/{name}/adapters", a.setAdapters)
mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
@@ -1635,6 +1643,10 @@ func (a *API) planModel(w http.ResponseWriter, r *http.Request) {
// the admin API answers only this machine and trusts whoever reaches it. Kept
// to a short printable name, so a claim cannot pass for more than one — no
// line breaks to forge a second log line, no length to bury the rest.
// Actor is who an admin API request says it comes from: the person who ran
// the command holding the admin token, as the CLI reports it.
func Actor(r *http.Request) string { return actorOf(r) }
func actorOf(r *http.Request) string {
claimed := r.Header.Get("X-Llmgw-Actor")
var b strings.Builder
internal/control/adminapi/audit.go
+77
View File
@@ -0,0 +1,77 @@
package adminapi
import (
"context"
"net/http"
"strconv"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// AuditReader lists recorded changes.
type AuditReader interface {
AuditEvents(ctx context.Context, f store.AuditFilter) ([]store.AuditEvent, error)
}
// SetAudit enables GET /api/v1/audit.
func (a *API) SetAudit(r AuditReader) { a.audit = r }
// audit answers GET /api/v1/audit?since=24h&actor=&contains=&limit=.
func (a *API) auditEvents(w http.ResponseWriter, r *http.Request) {
if a.audit == nil {
writeError(w, http.StatusNotImplemented, "this control plane keeps no audit log: it has no store")
return
}
q := r.URL.Query()
since := 7 * 24 * time.Hour
if s := q.Get("since"); s != "" {
d, err := parseSince(s)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
since = d
}
limit, _ := strconv.Atoi(q.Get("limit"))
ev, err := a.audit.AuditEvents(r.Context(), store.AuditFilter{
Since: time.Now().Add(-since), Actor: q.Get("actor"), Contains: q.Get("contains"), Limit: limit,
})
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
if ev == nil {
ev = []store.AuditEvent{}
}
writeJSON(w, http.StatusOK, ev)
}
// parseSince reads 90m, 24h, 7d.
func parseSince(s string) (time.Duration, error) {
if n, ok := cutSuffix(s, "d"); ok {
days, err := strconv.Atoi(n)
if err != nil || days < 0 {
return 0, errBadSince(s)
}
return time.Duration(days) * 24 * time.Hour, nil
}
d, err := time.ParseDuration(s)
if err != nil || d < 0 {
return 0, errBadSince(s)
}
return d, nil
}
type errBadSince string
func (e errBadSince) Error() string {
return "since " + string(e) + " is not a duration like 90m, 24h or 7d"
}
func cutSuffix(s, suf string) (string, bool) {
if len(s) > len(suf) && s[len(s)-len(suf):] == suf {
return s[:len(s)-len(suf)], true
}
return "", false
}
internal/control/adminapi/export.go
+96
View File
@@ -0,0 +1,96 @@
package adminapi
import (
"context"
"encoding/csv"
"fmt"
"io"
"net/http"
"strconv"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Exporter totals usage for an export.
type Exporter interface {
UsageExport(ctx context.Context, from, to time.Time, period string, by []string) ([]store.ExportRow, error)
}
// usageExport answers GET /api/v1/usage/export?from=2026-09-01&to=2026-10-01
// &period=day&by=project,model as CSV, or JSON with format=json.
func (a *API) usageExport(w http.ResponseWriter, r *http.Request) {
ex, ok := a.usage.(Exporter)
if !ok {
writeError(w, http.StatusNotImplemented, "this control plane keeps no usage records")
return
}
q := r.URL.Query()
day := func(s string, def time.Time) (time.Time, error) {
if s == "" {
return def, nil
}
if t, err := time.Parse("2006-01-02", s); err == nil {
return t, nil
}
return time.Parse(time.RFC3339, s)
}
now := time.Now().UTC()
from, err := day(q.Get("from"), time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.UTC))
if err != nil {
writeError(w, http.StatusBadRequest, "from: a date, 2026-09-01")
return
}
to, err := day(q.Get("to"), now)
if err != nil {
writeError(w, http.StatusBadRequest, "to: a date, 2026-10-01")
return
}
period := q.Get("period")
if period == "" {
period = "day"
}
var by []string
if b := q.Get("by"); b != "" {
by = strings.Split(b, ",")
}
rows, err := ex.UsageExport(r.Context(), from, to, period, by)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
if q.Get("format") == "json" {
writeJSON(w, http.StatusOK, rows)
return
}
w.Header().Set("Content-Type", "text/csv; charset=utf-8")
w.Header().Set("Content-Disposition", fmt.Sprintf(`attachment; filename="llmgw-usage-%s-%s.csv"`,
from.Format("20060102"), to.Format("20060102")))
WriteExportCSV(w, rows)
}
// WriteExportCSV writes rows with a header; cost in units as well as micros,
// so a spreadsheet needs no arithmetic.
func WriteExportCSV(w io.Writer, rows []store.ExportRow) {
c := csv.NewWriter(w)
_ = c.Write([]string{"period_start", "org", "team", "project", "user", "key", "model",
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros", "cost"})
for _, r := range rows {
_ = c.Write([]string{r.Period.Format(time.RFC3339), safe(r.Org), safe(r.Team), safe(r.Project), safe(r.User),
r.Key, safe(r.Model), i(r.Requests), i(r.Errors), i(r.Limited), i(r.Prompt), i(r.Completion),
i(r.CostMicros), strconv.FormatFloat(float64(r.CostMicros)/1e6, 'f', 6, 64)})
}
c.Flush()
}
func i(n int64) string { return strconv.FormatInt(n, 10) }
// safe keeps a name from being read as a formula by a spreadsheet: a cell
// starting with = + - @ is one in Excel and LibreOffice.
func safe(s string) string {
if s != "" && strings.ContainsRune("=+-@\t\r", rune(s[0])) {
return "'" + s
}
return s
}
internal/control/adminapi/guard_test.go
+59
View File
@@ -2,10 +2,15 @@ package adminapi
import (
"context"
"net/http"
"net/http/httptest"
"slices"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
type memGuard struct{ att []guard.Attachment }
@@ -57,3 +62,57 @@ func TestGuardPoliciesAreAttachedByTarget(t *testing.T) {
t.Errorf("detach again: %d", code)
}
}
type memAudit struct{ f store.AuditFilter }
func (m *memAudit) AuditEvents(_ context.Context, f store.AuditFilter) ([]store.AuditEvent, error) {
m.f = f
return []store.AuditEvent{{Actor: "ilker", Action: "POST /api/v1/models"}}, nil
}
func TestTheAuditLogIsReadWithFilters(t *testing.T) {
a, _ := setup(t, testModels())
m := &memAudit{}
a.SetAudit(m)
var ev []store.AuditEvent
if code := call(t, a, "GET", "/api/v1/audit?since=2d&actor=ilker&contains=models&limit=5", "", &ev); code != 200 || len(ev) != 1 {
t.Fatalf("%d %v", code, ev)
}
if m.f.Actor != "ilker" || m.f.Contains != "models" || m.f.Limit != 5 || time.Since(m.f.Since) < 47*time.Hour {
t.Errorf("filter %+v", m.f)
}
if code := call(t, a, "GET", "/api/v1/audit?since=soon", "", nil); code != 400 {
t.Errorf("a bad since: %d", code)
}
}
type memUsageExport struct {
memAudit
by []string
period string
}
func (m *memUsageExport) UsageByScope(context.Context, store.Scope, time.Time) ([]store.ScopeUsage, error) {
return nil, nil
}
func (m *memUsageExport) UsageExport(_ context.Context, from, to time.Time, period string, by []string) ([]store.ExportRow, error) {
m.by, m.period = by, period
return []store.ExportRow{{Period: from, Project: "=HYPERLINK(evil)", Model: "m", Requests: 3, CostMicros: 1500000}}, nil
}
func TestUsageIsExportedAsCSVThatASpreadsheetCannotRun(t *testing.T) {
a, _ := setup(t, testModels())
m := &memUsageExport{}
a.usage = m
mux := http.NewServeMux()
a.Register(mux)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, httptest.NewRequest("GET", "/api/v1/usage/export?from=2026-09-01&to=2026-10-01&period=month&by=project,model", nil))
body := rec.Body.String()
if rec.Code != 200 || !strings.HasPrefix(rec.Header().Get("Content-Type"), "text/csv") ||
!strings.Contains(body, "period_start,org,team,project") || !strings.Contains(body, "'=HYPERLINK") ||
!strings.Contains(body, "1.500000") || m.period != "month" || !slices.Equal(m.by, []string{"project", "model"}) {
t.Errorf("%d %s %+v", rec.Code, body, m)
}
}
internal/control/adminapi/notify.go
+19
View File
@@ -0,0 +1,19 @@
package adminapi
import (
"net/http"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
)
// SetNotifier gives the API the webhooks `ctl notify test` reaches.
func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
// notifyTest sends a test event to every webhook and says how each answered.
func (a *API) notifyTest(w http.ResponseWriter, r *http.Request) {
if a.notifier == nil {
writeError(w, http.StatusNotFound, "no webhooks: server.yaml has no notify.webhooks")
return
}
writeJSON(w, http.StatusOK, a.notifier.Test(r.Context()))
}
internal/control/modelserve/modelserve.go
+4
View File
@@ -31,6 +31,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
)
// Registry reads a repository.
@@ -85,6 +86,9 @@ type Deps struct {
// UnplacedGrace is how long a model may go unplaced before the job
// stops waiting for it: a minute unless set.
UnplacedGrace time.Duration
// Notify is told when a model's repository publishes new weights; nil
// tells no one.
Notify *notify.Notifier
}
// errBusy is a model the cluster has the hardware for, all of it in use now.
internal/control/modelserve/updates.go
+9
View File
@@ -16,6 +16,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"github.com/prometheus/client_golang/prometheus"
)
@@ -343,8 +344,16 @@ func (s *Server) checkOne(ctx context.Context, m catalog.Model, o *catalog.Origi
}
}
s.mu.Lock()
prev, had := s.updates[m.Name]
s.updates[m.Name] = u
s.mu.Unlock()
// Told once per new revision, not at every check that finds it still there.
if u.Available && (!had || !prev.Available || prev.Latest != u.Latest) {
s.deps.Notify.Notify(notify.UpdateAvailable, m.Name+"@"+u.Latest,
fmt.Sprintf("%s has new weights at %s (%s); `llmgw ctl model update %s` moves to them",
m.Name, o.Repo, catalog.ShortRevision(u.Latest), m.Name),
map[string]any{"model": m.Name, "repo": o.Repo, "current": o.Revision, "latest": u.Latest})
}
return u
}
internal/control/modelserve/updates_test.go
+14
View File
@@ -14,6 +14,8 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify/notifytest"
)
const (
@@ -338,3 +340,15 @@ func TestAnOriginGivenByHandIsChecked(t *testing.T) {
t.Fatalf("%+v", o)
}
}
func TestNewWeightsAreToldOncePerRevision(t *testing.T) {
s, _, _, _, _ := updSetup(t)
n, sent := notifytest.Catch(t)
s.deps.Notify = n
s.CheckUpdates(context.Background())
s.CheckUpdates(context.Background()) // still there: not told again
got := sent()
if len(got) != 1 || got[0].Kind != notify.UpdateAvailable || got[0].Detail["model"] != "qwen" {
t.Fatalf("sent %+v, want one update_available for qwen", got)
}
}
internal/control/reconciler/notify_test.go
+46
View File
@@ -0,0 +1,46 @@
package reconciler
import (
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify/notifytest"
)
func TestParkingAModelTellsSomeone(t *testing.T) {
h := newHarness(t, testModel(1), gpuNode("1", 8))
n, sent := notifytest.Catch(t)
h.r.SetNotifier(n)
failRepeatedly(t, h, DefaultConfig().ParkAfter, "unknown model architecture: 'ggmlc'")
h.advance(DefaultConfig().MaxBackoff + time.Second)
h.r.Reconcile(t.Context())
got := sent()
if len(got) != 1 || got[0].Kind != notify.ModelParked || got[0].Subject != "llama" {
t.Fatalf("sent %+v, want one model_parked for llama", got)
}
}
func TestLosingANodeTellsSomeone(t *testing.T) {
h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8))
n, sent := notifytest.Catch(t)
h.r.SetNotifier(n)
h.r.Reconcile(t.Context())
h.markReady()
h.r.Reconcile(t.Context())
for _, d := range []time.Duration{20 * time.Second, 90 * time.Second} {
h.clock.advance(d)
touchAt(h.inv, "2", h.clock.now())
h.r.Reconcile(t.Context())
}
if n, _ := h.inv.Get("1"); n.State != inventory.Lost {
t.Fatalf("node 1 is %s", n.State)
}
got := sent()
if len(got) != 1 || got[0].Kind != notify.NodeLost || got[0].Subject != "1" {
t.Fatalf("sent %+v, want one node_lost for 1", got)
}
}
internal/control/reconciler/reconciler.go
+18
View File
@@ -8,6 +8,7 @@ import (
"math/rand/v2"
"slices"
"sync"
"sync/atomic"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
@@ -16,6 +17,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
)
// ShardCommand is everything an agent needs to launch one shard.
@@ -231,6 +233,9 @@ type Reconciler struct {
disp Dispatcher
log *slog.Logger
now func() time.Time
// notify tells people about parked models and lost nodes; nil tells
// no one.
notify atomic.Pointer[notify.Notifier]
mu sync.Mutex
replicas map[string]*Replica
@@ -559,6 +564,9 @@ func (r *Reconciler) expireNodes() {
r.inv.SetState(n.ID, inventory.Lost)
r.log.Error("node lost; its replicas will be rebuilt elsewhere",
"node", n.ID, "silent_for", silent.Round(time.Second))
r.notify.Load().Notify(notify.NodeLost, n.ID,
fmt.Sprintf("node %s was lost: silent for %s; its replicas are rebuilt elsewhere", n.ID, silent.Round(time.Second)),
map[string]any{"node": n.ID, "silent_seconds": int(silent.Seconds())})
r.failReplicasOn(n.ID, "node "+n.ID+" was lost")
}
}
@@ -644,6 +652,10 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
r.log.Error("model parked: it failed every time it started, and is not retried "+
"until its definition changes or `llmgw ctl retry` is run",
"model", rep.Model, "failures", n, "last_error", rep.Err)
r.notify.Load().Notify(notify.ModelParked, rep.Model,
fmt.Sprintf("model %s is parked: it failed %d times in a row (%s); `llmgw ctl retry %s` tries again",
rep.Model, n, rep.Err, rep.Model),
map[string]any{"model": rep.Model, "failures": n, "last_error": rep.Err})
}
}
}
@@ -1216,3 +1228,9 @@ func (r *Reconciler) engineKey(replicaID string) string {
}
return r.cfg.EngineKey(replicaID)
}
// SetNotifier gives the reconciler somewhere to say a model was parked or a
// node lost.
func (r *Reconciler) SetNotifier(n *notify.Notifier) {
r.notify.Store(n)
}
internal/docgen/reference_gen.go
+30
View File
@@ -27,6 +27,9 @@ var Reference = []Block{
{Key: "default_rpm", Type: "int", Doc: "DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.\n\nUnlimited used to be the default, and a project created without limits is\nthe easy path — the \"default\" project on this cluster still is one. The\nfirst sign of that is the bill or the outage. Zero keeps the old\nbehaviour for anyone who wants it, but it should be a choice rather than\nan omission. A fallback, not a ceiling: an explicit limit at any scope\nwins, above or below.", Nested: ""},
{Key: "default_tpm", Type: "int", Doc: "", Nested: ""},
{Key: "signup_open", Type: "bool", Doc: "SignupOpen serves the signup form. Accounts it creates are pending until\nan operator approves them, so this is not a way in — but turning it off\nmeans accounts are made with `llmgw users create` instead.", Nested: ""},
{Key: "oidc", Type: "OIDC", Doc: "OIDC is sign-in through an identity provider.", Nested: "config.OIDC"},
{Key: "scim", Type: "SCIM", Doc: "SCIM is account provisioning by the identity provider.", Nested: "config.SCIM"},
{Key: "max_key_ttl", Type: "time.Duration", Doc: "MaxKeyTTL is how long any API key may live; a key asked for with no\nexpiry is given this. Organizations, teams and projects can set\nshorter ones (llmgw keys ttl). Zero is no limit.", Nested: ""},
{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
{Key: "pricing", Type: "map[string]Price", Doc: "Pricing is what each model costs, for budgets. A model absent from the\ntable costs whatever DefaultPrice says, which is nothing unless set.", Nested: ""},
{Key: "default_price", Type: "Price", Doc: "", Nested: "config.Price"},
@@ -125,6 +128,19 @@ var Reference = []Block{
{Key: "window", Type: "time.Duration", Doc: "Window is how far back the charts look. Zero takes an hour.", Nested: ""},
{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds one query. It runs while somebody waits for a page, so a\nmonitoring system that is slow today is skipped rather than waited for.", Nested: ""},
}},
{Package: "config", Name: "OIDC", Title: "OIDC", Doc: "", Fields: []Field{
{Key: "issuer", Type: "string", Doc: "Issuer is the provider's address; its /.well-known/openid-configuration\nsays the rest.", Nested: ""},
{Key: "client_id", Type: "string", Doc: "ClientID and the file holding its secret, as registered with the\nprovider.", Nested: ""},
{Key: "client_secret_file", Type: "string", Doc: "", Nested: ""},
{Key: "redirect_url", Type: "string", Doc: "RedirectURL is where the provider sends people back:\nhttps://<gateway>/login/oidc/callback. Empty works it out from the\nrequest, which is right unless a proxy changes the address.", Nested: ""},
{Key: "label", Type: "string", Doc: "Label is what the sign-in button says: \"Sign in with <label>\".", Nested: ""},
{Key: "scopes", Type: "[]string", Doc: "Scopes asked for besides openid; empty is email and profile, and\ngroups when Roles or Memberships use them.", Nested: ""},
{Key: "groups_claim", Type: "string", Doc: "GroupsClaim names the ID token claim holding groups; empty is\n\"groups\".", Nested: ""},
{Key: "create_accounts", Type: "bool", Doc: "CreateAccounts makes an account, active, for a person the provider\nvouches for and the gateway does not know. Off, only existing accounts\nsign in this way.", Nested: ""},
{Key: "roles", Type: "map[string][]string", Doc: "Roles maps a staff role (viewer, operator, billing, admin) to the\ngroups that give it. Set at every sign-in: someone taken out of the\ngroup loses the role the next time.", Nested: ""},
{Key: "memberships", Type: "map[string]string", Doc: "Memberships maps a group to a place in the tree — org:NAME,\nteam:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in.", Nested: ""},
{Key: "no_passwords", Type: "bool", Doc: "PasswordSignIn off refuses passwords on the sign-in form: people come\nin through the provider only. Administrators still can, so a broken\nprovider does not lock everybody out.", Nested: ""},
}},
{Package: "discovery", Name: "Options", Title: "Options", Doc: "", Fields: []Field{
{Key: "nvidia_smi", Type: "string", Doc: "NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.\nWorth configuring on nodes where the driver lives outside the default\npaths, which containerized driver installs often do.", Nested: ""},
{Key: "ib_root", Type: "string", Doc: "IBRoot is the sysfs InfiniBand directory.", Nested: ""},
@@ -226,6 +242,9 @@ var Reference = []Block{
{Key: "deny", Type: "[]Pattern", Doc: "Deny are patterns, matched against the text as a model reads it.", Nested: ""},
{Key: "max_chars", Type: "int", Doc: "MaxChars refuses text longer than this; zero is no limit.", Nested: ""},
}},
{Package: "config", Name: "SCIM", Title: "SCIM", Doc: "", Fields: []Field{
{Key: "token_file", Type: "string", Doc: "TokenFile holds the bearer token the provider sends. Empty turns\nprovisioning off.", Nested: ""},
}},
{Package: "engines", Name: "SGLangConfig", Title: "SGLangConfig", Doc: "", Fields: []Field{
{Key: "python", Type: "string", Doc: "Python is the interpreter of the SGLang virtualenv. SGLang is launched as\na module rather than a console script, which is the documented entry\npoint and avoids depending on the script being on PATH.", Nested: ""},
}},
@@ -265,6 +284,7 @@ var Reference = []Block{
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
{Key: "guard", Type: "guard.Config", Doc: "Guard is the library of guardrail policies: what requests and answers\nare checked for, and what is done. Applied to every request through\nguard.default, and to parts of the tenancy tree by attaching them\n(llmgw ctl guard attach). Empty checks nothing.", Nested: "guard.Config"},
{Key: "notify", Type: "notify.Config", Doc: "Notify sends what needs a person — a model parked, a node lost, a\nbudget spent, new weights published — to webhooks, signed, without a\nPrometheus and Alertmanager in between. Empty sends nothing.", Nested: "notify.Config"},
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
@@ -304,6 +324,12 @@ var Reference = []Block{
{Key: "launch_mode", Type: "string", Doc: "LaunchMode selects the multi-node mechanism: \"mp\", the default, or\n\"ray\".\n\nmp is what vLLM documents today — every node runs serve with its rank,\nthe workers headless — and needs nothing but vLLM. ray is the older way,\nkept for a vLLM too old for mp's multi-node flags; it needs Ray on every\nnode and has not been checked against a real cluster.", Nested: ""},
{Key: "ray_bin", Type: "string", Doc: "RayBin is used when LaunchMode is \"ray\".", Nested: ""},
}},
{Package: "notify", Name: "Webhook", Title: "Webhook", Doc: "", Fields: []Field{
{Key: "url", Type: "string", Doc: "", Nested: ""},
{Key: "format", Type: "string", Doc: "Format is json (the event, with X-Llmgw-Signature when SecretFile is\nset) or slack ({\"text\": …}, which Slack, Mattermost and Rocket.Chat\nincoming webhooks take).", Nested: ""},
{Key: "events", Type: "[]string", Doc: "Events are the kinds sent here; empty is all.", Nested: ""},
{Key: "secret_file", Type: "string", Doc: "SecretFile holds the key the body is signed with: X-Llmgw-Signature\nis sha256=<hex HMAC of the body>.", Nested: ""},
}},
{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
@@ -362,6 +388,10 @@ var Reference = []Block{
{Key: "default", Type: "[]string", Doc: "", Nested: ""},
{Key: "policies", Type: "map[string]Policy", Doc: "", Nested: ""},
}},
{Package: "notify", Name: "Config", Title: "notify.Config", Doc: "", Fields: []Field{
{Key: "webhooks", Type: "[]Webhook", Doc: "", Nested: ""},
{Key: "host", Type: "string", Doc: "Host names the gateway in messages; empty is this machine's name.", Nested: ""},
}},
{Package: "supervisor", Name: "Config", Title: "supervisor.Config", Doc: "", Fields: []Field{
{Key: "work_dir", Type: "string", Doc: "WorkDir holds pid files and per-shard scratch space.", Nested: ""},
{Key: "log_lines", Type: "int", Doc: "LogLines is how many lines of engine output to retain per shard.", Nested: ""},
internal/docgen/render.go
+2
View File
@@ -28,6 +28,8 @@ var Sources = []string{
"internal/accounting/cost.go",
"internal/guard/policy.go",
"internal/guard/judge.go",
"internal/config/oidc.go",
"internal/notify/notify.go",
}
// notReference are files with koanf tags that an operator never writes.
internal/guard/gate.go
+13
View File
@@ -40,6 +40,8 @@ type Gate struct {
tools toolMemo
judger *judge
m *gateMetrics
// alarm is told each time a judge gives no verdict; nil tells no one.
alarm func(model string, err error)
mu sync.Mutex
attached map[string][]string // "scope/id" → policy names
@@ -363,6 +365,7 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
res, err := g.judger.ask(ctx, jp, user, "")
if err != nil {
g.log.Warn("the judge gave no verdict on a request", "judge", jp.Model, "err", err)
g.judgeFailed(jp.Model, err)
}
if err != nil && p.Fail == "closed" && !v.Shadow {
v.Took = g.now().Sub(start)
@@ -402,6 +405,16 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
// judgeFinding is what a judge's answer, or its failing to give one, comes
// to under a policy.
// SetJudgeAlarm says whom to tell when a judge gives no verdict: set before
// the gate serves.
func (g *Gate) SetJudgeAlarm(f func(model string, err error)) { g.alarm = f }
func (g *Gate) judgeFailed(model string, err error) {
if g.alarm != nil {
g.alarm(model, err)
}
}
func judgeFinding(p Policy, action string, res judgment, err error) finding {
if err != nil {
// Unanswered: closed stops it, open lets it through, said.
internal/guard/judge_test.go
+5
View File
@@ -105,9 +105,14 @@ func TestAJudgeThatDoesNotAnswerFailsAsThePolicySays(t *testing.T) {
t.Errorf("fail open: %+v %v after %s", v, err, time.Since(start))
}
closed := judged(t, Policy{Fail: "closed", Judge: &JudgePolicy{Model: "guard", Input: "block"}}, &fakeJudge{err: errors.New("down")})
var alarmed []string
closed.SetJudgeAlarm(func(model string, _ error) { alarmed = append(alarmed, model) })
if _, err := closed.Input(ctx, Subject{}, chat("user", "hi")); err == nil {
t.Error("fail closed served unjudged")
}
if len(alarmed) != 1 || alarmed[0] != "guard" {
t.Errorf("alarm told %v", alarmed)
}
// A judge that is not a guard model is a judge that cannot be asked.
bad := judged(t, Policy{Fail: "closed", Judge: &JudgePolicy{Model: "chat-model", Input: "block"}}, &fakeJudge{})
if _, err := bad.Input(ctx, Subject{}, chat("user", "hi")); err == nil {
internal/guard/output.go
+1
View File
@@ -194,6 +194,7 @@ func (f *outFilter) askJudge(text string, final bool) finding {
f.took += time.Since(start)
if err != nil {
f.g.log.Warn("the judge gave no verdict on an answer", "judge", jp.Model, "err", err)
f.g.judgeFailed(jp.Model, err)
}
jf := judgeFinding(f.p, jp.Output, res, err)
if jf.failOpen {
internal/notify/notify.go
+292
View File
@@ -0,0 +1,292 @@
// Package notify tells the people running the gateway about what needs them,
// without a Prometheus in between: a model parked after failing every start,
// a node lost, a budget spent, a model's repository with new weights, a
// guardrails judge not answering. Each
// event goes to the webhooks configured for it — JSON signed with a shared
// secret, or a Slack-compatible message.
//
// Sending never holds up what raised the event: events are queued, and a
// webhook that does not answer is retried a few times and then given up on.
// The same event about the same thing is sent once an hour at most.
package notify
import (
"bytes"
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"net/url"
"os"
"slices"
"strings"
"sync"
"time"
)
// Event kinds.
const (
ModelParked = "model_parked"
NodeLost = "node_lost"
BudgetSpent = "budget_spent"
UpdateAvailable = "update_available"
// JudgeUnavailable is a guardrails judge giving no verdict: with
// fail: closed, requests are being refused; with open, served unjudged.
JudgeUnavailable = "judge_unavailable"
)
// Kinds are every event there is.
var Kinds = []string{ModelParked, NodeLost, BudgetSpent, UpdateAvailable, JudgeUnavailable}
// Event is one thing that happened.
type Event struct {
Kind string `json:"kind"`
At time.Time `json:"at"`
Subject string `json:"subject"`
Message string `json:"message"`
Detail map[string]any `json:"detail,omitempty"`
}
// Webhook is where events go.
type Webhook struct {
URL string `koanf:"url"`
// Format is json (the event, with X-Llmgw-Signature when SecretFile is
// set) or slack ({"text": …}, which Slack, Mattermost and Rocket.Chat
// incoming webhooks take).
Format string `koanf:"format"`
// Events are the kinds sent here; empty is all.
Events []string `koanf:"events"`
// SecretFile holds the key the body is signed with: X-Llmgw-Signature
// is sha256=<hex HMAC of the body>.
SecretFile string `koanf:"secret_file"`
}
// Config is server.yaml's notify: section.
type Config struct {
Webhooks []Webhook `koanf:"webhooks"`
// Host names the gateway in messages; empty is this machine's name.
Host string `koanf:"host"`
}
// Validate checks the settings.
func (c Config) Validate() error {
for i, w := range c.Webhooks {
if !strings.HasPrefix(w.URL, "https://") && !strings.HasPrefix(w.URL, "http://") {
return fmt.Errorf("notify.webhooks[%d].url %q: an http(s) address", i, w.URL)
}
switch w.Format {
case "", "json", "slack":
default:
return fmt.Errorf("notify.webhooks[%d].format %q: json or slack", i, w.Format)
}
for _, e := range w.Events {
if !slices.Contains(Kinds, e) {
return fmt.Errorf("notify.webhooks[%d].events: %q is not one of %s", i, e, strings.Join(Kinds, ", "))
}
}
}
return nil
}
// Notifier sends events. A nil one sends nothing.
type Notifier struct {
hooks []hook
host string
log *slog.Logger
client *http.Client
queue chan Event
retry time.Duration
mu sync.Mutex
sent map[string]time.Time
wg sync.WaitGroup
}
type hook struct {
Webhook
secret []byte
}
// dedupe is how long the same event about the same thing is not sent again.
const dedupe = time.Hour
// New builds a notifier and starts sending; it stops when ctx ends. Nil when
// nothing is configured.
func New(ctx context.Context, c Config, log *slog.Logger) (*Notifier, error) {
if len(c.Webhooks) == 0 {
return nil, nil
}
if err := c.Validate(); err != nil {
return nil, err
}
if log == nil {
log = slog.Default()
}
n := &Notifier{host: c.Host, log: log, client: &http.Client{Timeout: 10 * time.Second},
queue: make(chan Event, 256), sent: map[string]time.Time{}, retry: 5 * time.Second}
if n.host == "" {
n.host, _ = os.Hostname()
}
for _, w := range c.Webhooks {
h := hook{Webhook: w}
if w.SecretFile != "" {
b, err := os.ReadFile(w.SecretFile)
if err != nil {
return nil, fmt.Errorf("notify: %w", err)
}
h.secret = bytes.TrimSpace(b)
}
n.hooks = append(n.hooks, h)
}
n.wg.Add(1)
go n.run(ctx)
return n, nil
}
// Notify queues an event. It never blocks: with the queue full the event is
// dropped, and said so in the log.
func (n *Notifier) Notify(kind, subject, message string, detail map[string]any) {
if n == nil {
return
}
k := kind + "\x00" + subject
now := time.Now()
n.mu.Lock()
if last, ok := n.sent[k]; ok && now.Sub(last) < dedupe {
n.mu.Unlock()
return
}
n.sent[k] = now
for key, at := range n.sent {
if now.Sub(at) > dedupe {
delete(n.sent, key)
}
}
n.mu.Unlock()
select {
case n.queue <- Event{Kind: kind, At: now.UTC(), Subject: subject, Message: message, Detail: detail}:
default:
n.log.Warn("a notification was dropped: too many waiting", "kind", kind, "subject", subject)
}
}
// Wait waits for what is queued to be sent, for tests and shutdown.
func (n *Notifier) Wait() {
if n == nil {
return
}
close(n.queue)
n.wg.Wait()
}
func (n *Notifier) run(ctx context.Context) {
defer n.wg.Done()
for {
select {
case <-ctx.Done():
return
case e, ok := <-n.queue:
if !ok {
return
}
for _, h := range n.hooks {
if len(h.Events) > 0 && !slices.Contains(h.Events, e.Kind) {
continue
}
if err := n.send(ctx, h, e, 3); err != nil {
n.log.Warn("a notification could not be delivered", "kind", e.Kind, "subject", e.Subject, "err", err)
}
}
}
}
}
// Test sends a test event to every webhook now, once each, and says how each
// answered: what `llmgw ctl notify test` shows.
func (n *Notifier) Test(ctx context.Context) []Result {
if n == nil {
return nil
}
e := Event{Kind: "test", At: time.Now().UTC(), Subject: "test",
Message: "a test notification: this webhook is reached"}
var out []Result
for _, h := range n.hooks {
r := Result{URL: redact(h.URL)}
if err := n.send(ctx, h, e, 1); err != nil {
r.Err = err.Error()
}
out = append(out, r)
}
return out
}
// Result is how one webhook answered a test.
type Result struct {
URL string `json:"url"`
Err string `json:"error,omitempty"`
}
// redact keeps a webhook's host and drops its path, where Slack and others
// put the secret.
func redact(u string) string {
if i := strings.Index(u, "://"); i >= 0 {
if j := strings.IndexByte(u[i+3:], '/'); j >= 0 {
return u[:i+3+j] + "/…"
}
}
return u
}
func (n *Notifier) send(ctx context.Context, h hook, e Event, attempts int) error {
var body []byte
if h.Format == "slack" {
body, _ = json.Marshal(map[string]string{"text": fmt.Sprintf("*llmgw on %s* — %s", n.host, e.Message)})
} else {
ev := e
if ev.Detail == nil {
ev.Detail = map[string]any{}
}
ev.Detail["gateway"] = n.host
body, _ = json.Marshal(ev)
}
for attempt := 1; ; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, h.URL, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("X-Llmgw-Event", e.Kind)
if h.secret != nil {
m := hmac.New(sha256.New, h.secret)
m.Write(body)
req.Header.Set("X-Llmgw-Signature", "sha256="+hex.EncodeToString(m.Sum(nil)))
}
resp, err := n.client.Do(req)
// Not the error as it comes: it names the whole address, and the
// path of a Slack or PagerDuty webhook is its secret.
var ue *url.Error
if errors.As(err, &ue) {
err = fmt.Errorf("%s %s: %w", ue.Op, redact(h.URL), ue.Err)
}
if err == nil {
resp.Body.Close()
if resp.StatusCode < 300 {
return nil
}
err = fmt.Errorf("answered %s", resp.Status)
}
if attempt >= attempts {
return err
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(n.retry * time.Duration(attempt)):
}
}
}
internal/notify/notify_test.go
+169
View File
@@ -0,0 +1,169 @@
package notify
import (
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
type got struct {
mu sync.Mutex
calls []*http.Request
body [][]byte
}
func (g *got) handler(fail int) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
b, _ := io.ReadAll(r.Body)
g.mu.Lock()
defer g.mu.Unlock()
g.calls = append(g.calls, r)
g.body = append(g.body, b)
if len(g.calls) <= fail {
w.WriteHeader(http.StatusBadGateway)
}
}
}
func newNotifier(t *testing.T, c Config) *Notifier {
t.Helper()
n, err := New(context.Background(), c, nil)
if err != nil {
t.Fatal(err)
}
n.retry = time.Millisecond
return n
}
func TestSignedJSON(t *testing.T) {
var g got
srv := httptest.NewServer(g.handler(0))
defer srv.Close()
secret := filepath.Join(t.TempDir(), "s")
os.WriteFile(secret, []byte("k3y\n"), 0o600)
n := newNotifier(t, Config{Host: "gw1", Webhooks: []Webhook{{URL: srv.URL, SecretFile: secret}}})
n.Notify(ModelParked, "qwen", "model qwen is parked", map[string]any{"failures": 3})
n.Wait()
if len(g.calls) != 1 {
t.Fatalf("%d calls, want 1", len(g.calls))
}
m := hmac.New(sha256.New, []byte("k3y"))
m.Write(g.body[0])
if want := "sha256=" + hex.EncodeToString(m.Sum(nil)); g.calls[0].Header.Get("X-Llmgw-Signature") != want {
t.Errorf("signature %q, want %q", g.calls[0].Header.Get("X-Llmgw-Signature"), want)
}
if g.calls[0].Header.Get("X-Llmgw-Event") != ModelParked {
t.Errorf("event header %q", g.calls[0].Header.Get("X-Llmgw-Event"))
}
var e Event
if err := json.Unmarshal(g.body[0], &e); err != nil {
t.Fatal(err)
}
if e.Kind != ModelParked || e.Subject != "qwen" || e.Detail["gateway"] != "gw1" || e.Detail["failures"] != float64(3) {
t.Errorf("event %+v", e)
}
}
func TestSlackFiltersAndDedupes(t *testing.T) {
var g got
srv := httptest.NewServer(g.handler(0))
defer srv.Close()
n := newNotifier(t, Config{Host: "gw1", Webhooks: []Webhook{{URL: srv.URL, Format: "slack", Events: []string{NodeLost}}}})
n.Notify(ModelParked, "qwen", "not for this hook", nil)
n.Notify(NodeLost, "n1", "node n1 was lost", nil)
n.Notify(NodeLost, "n1", "node n1 was lost again", nil) // within the hour: not sent
n.Notify(NodeLost, "n2", "node n2 was lost", nil)
n.Wait()
if len(g.calls) != 2 {
t.Fatalf("%d calls, want 2: %q", len(g.calls), g.body)
}
var m map[string]string
json.Unmarshal(g.body[0], &m)
if !strings.Contains(m["text"], "gw1") || !strings.Contains(m["text"], "node n1 was lost") {
t.Errorf("text %q", m["text"])
}
if g.calls[0].Header.Get("X-Llmgw-Signature") != "" {
t.Error("signed without a secret")
}
}
func TestRetries(t *testing.T) {
var g got
srv := httptest.NewServer(g.handler(2))
defer srv.Close()
n := newNotifier(t, Config{Webhooks: []Webhook{{URL: srv.URL}}})
n.Notify(BudgetSpent, "org:1", "spent", nil)
n.Wait()
if len(g.calls) != 3 {
t.Fatalf("%d attempts, want 3 (two failures, then delivered)", len(g.calls))
}
var h got
srv2 := httptest.NewServer(h.handler(10))
defer srv2.Close()
n = newNotifier(t, Config{Webhooks: []Webhook{{URL: srv2.URL}}})
n.Notify(BudgetSpent, "org:1", "spent", nil)
n.Wait()
if len(h.calls) != 3 {
t.Fatalf("%d attempts at a failing hook, want 3 then given up", len(h.calls))
}
}
func TestNilAndValidate(t *testing.T) {
var n *Notifier
n.Notify(NodeLost, "x", "y", nil) // must not panic
n.Wait()
if n, err := New(context.Background(), Config{}, nil); n != nil || err != nil {
t.Fatalf("nothing configured: %v %v", n, err)
}
for _, c := range []Config{
{Webhooks: []Webhook{{URL: "ftp://x"}}},
{Webhooks: []Webhook{{URL: "https://x", Format: "teams"}}},
{Webhooks: []Webhook{{URL: "https://x", Events: []string{"model_gone"}}}},
} {
if c.Validate() == nil {
t.Errorf("%+v accepted", c)
}
}
if _, err := New(context.Background(), Config{Webhooks: []Webhook{{URL: "https://x", SecretFile: "/nonexistent"}}}, nil); err == nil {
t.Error("a missing secret file accepted")
}
}
func TestTestSaysHowEachAnsweredWithoutTheSecretPath(t *testing.T) {
var g got
ok := httptest.NewServer(g.handler(0))
defer ok.Close()
bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusForbidden) }))
defer bad.Close()
n := newNotifier(t, Config{Webhooks: []Webhook{
{URL: ok.URL + "/services/T0/B0/s3cret"},
{URL: bad.URL + "/hook"},
{URL: "http://127.0.0.1:1/services/T0/B0/s3cret"},
}})
defer n.Wait()
res := n.Test(context.Background())
if len(res) != 3 || res[0].Err != "" || !strings.Contains(res[1].Err, "403") || res[2].Err == "" {
t.Fatalf("%+v", res)
}
for _, r := range res {
if strings.Contains(r.URL+r.Err, "s3cret") {
t.Errorf("the secret path shown: %+v", r)
}
}
}
internal/notify/notifytest/notifytest.go
+41
View File
@@ -0,0 +1,41 @@
// Package notifytest catches notifications in tests.
package notifytest
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"sync"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
)
// Catch returns a notifier whose webhook records what it is sent, and a func
// that waits for the sending and returns the events. The notifier takes no
// more events once that func is called.
func Catch(t *testing.T) (*notify.Notifier, func() []notify.Event) {
t.Helper()
var mu sync.Mutex
var got []notify.Event
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
b, _ := io.ReadAll(req.Body)
var e notify.Event
_ = json.Unmarshal(b, &e)
mu.Lock()
got = append(got, e)
mu.Unlock()
}))
t.Cleanup(srv.Close)
n, err := notify.New(t.Context(), notify.Config{Webhooks: []notify.Webhook{{URL: srv.URL}}}, nil)
if err != nil {
t.Fatal(err)
}
return n, func() []notify.Event {
n.Wait()
mu.Lock()
defer mu.Unlock()
return got
}
}
internal/router/activate.go
+10
View File
@@ -3,6 +3,8 @@ package router
import (
"context"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
)
// Activator loads a model that is configured but not running.
@@ -90,3 +92,11 @@ func (r *Router) activate(ctx context.Context, model string) bool {
// own timeout is not what ends the request — being told to retry is a better
// outcome than a socket closing with nothing on it.
const defaultActivationWait = 45 * time.Second
// SetNotifier tells the accountant where to say a budget is spent. Nothing
// without a store, where there are no budgets.
func (r *Router) SetNotifier(n *notify.Notifier) {
if r.acct != nil {
r.acct.SetNotifier(n)
}
}
internal/router/guard_test.go
+13
View File
@@ -255,3 +255,16 @@ func TestAJudgeModelIsNotNamedToClientsInErrors(t *testing.T) {
t.Errorf("named: %s", rec.Body.String())
}
}
func TestSignInThroughAProviderNeedsNoAPIKey(t *testing.T) {
e := newFakeEngine(t, "qwen2.5-14b-instruct")
rt, _ := pricedRouter(t, e.srv.URL)
rt.oidc = config.OIDC{Issuer: "http://127.0.0.1:1", ClientID: "llmgw"}
rec := httptest.NewRecorder()
rt.Handler().ServeHTTP(rec, httptest.NewRequest("GET", "/login/oidc", nil))
// The provider is not there, so the page says so — but it is the page,
// not the API's "no key".
if rec.Code == http.StatusUnauthorized || !strings.Contains(rec.Body.String(), "sign-in service") {
t.Errorf("%d %s", rec.Code, rec.Body.String())
}
}
internal/router/handlers.go
+19 -1
View File
@@ -10,6 +10,7 @@ import (
"fmt"
"io"
"net/http"
"os"
"slices"
"strconv"
"strings"
@@ -23,6 +24,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
"gitea.mynodes.xyz/ilker/llmgw/internal/scim"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
@@ -180,6 +182,22 @@ func (r *Router) Handler() http.Handler {
pages.SetPersonalOff(r.personalOff)
pages.SetLlamaCppVersion(r.llamaVersion)
pages.SetPersonalDefaults(r.personalDefaults)
if r.store != nil {
pages.SetAudit(r.store)
if r.oidc.Enabled() {
if err := pages.RegisterOIDC(mux, r.oidc, r.store); err != nil {
r.log.Error("single sign-on is off", "err", err)
}
}
if r.scim.TokenFile != "" {
if b, err := os.ReadFile(r.scim.TokenFile); err != nil || strings.TrimSpace(string(b)) == "" {
r.log.Error("provisioning (SCIM) is off: its token cannot be read", "file", r.scim.TokenFile, "err", err)
} else {
scim.New(r.store, strings.TrimSpace(string(b)),
scim.Mapping{Roles: r.oidc.Roles, Memberships: r.oidc.Memberships}, r.log).Register(mux)
}
}
}
if pol, ok := r.guard.(interface {
Policy(guard.Subject) (guard.Policy, string)
}); ok {
@@ -253,7 +271,7 @@ func (r *Router) Handler() http.Handler {
// stays open because a load balancer has no key.
return auth.Middleware(r.auth, r.log,
"/health", "/ready", "/leader", "/", "/about", "/_static/", "/claim/",
"/signup", "/login", "/logout", "/profile", "/profile/",
"/signup", "/login", "/login/", "/logout", "/profile", "/profile/", "/scim/",
"/admin", "/admin/", "/chat", "/chat/", "/org", "/org/", "/setup/")(mux)
}
internal/router/router.go
+6
View File
@@ -93,6 +93,10 @@ type Router struct {
// is the policies it was built from, for the pages that show them.
guard guard.Guard
guardLibrary guard.Config
// oidc is sign-in through an identity provider, and scim provisioning
// by it.
oidc config.OIDC
scim config.SCIM
// threads holds conversations and memory notes. It is store when auth is on,
// its own file when only the threads block is configured, and nil when
@@ -278,6 +282,8 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
st.SetDefaultLimits(defRPM, defTPM)
st.SetPersonalDefaults(personalLimits(cfg.Auth.Personal))
r.personalOff = cfg.Auth.Personal.Disabled
r.oidc = cfg.Auth.OIDC
r.scim = cfg.Auth.SCIM
r.personalDefaults = personalLimits(cfg.Auth.Personal)
r.llamaVersion = cfg.WantedEngines()["llamacpp"]
internal/scim/scim.go
+609
View File
@@ -0,0 +1,609 @@
// Package scim lets an identity provider keep the gateway's accounts in
// step with its own (SCIM 2.0, RFC 7643/7644): people made, renamed,
// suspended and reinstated as they are there, and the groups that give
// staff roles and places in the tree followed as people join and leave them.
//
// What it takes of SCIM is what Entra ID, Okta and Keycloak send: Users by
// userName (the email address), active, name; Groups by displayName, their
// members changed with PATCH. Deleting a user suspends the account: its keys,
// usage and history stay, and reinstating it gives it all back.
package scim
import (
"context"
"crypto/rand"
"crypto/subtle"
"encoding/base32"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"regexp"
"slices"
"sort"
"strconv"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Store is what provisioning needs.
type Store interface {
Users(ctx context.Context, state store.UserState) ([]store.User, error)
UserByID(ctx context.Context, id string) (store.User, error)
UserByEmail(ctx context.Context, email string) (store.User, error)
CreateUser(ctx context.Context, u store.User) error
SetUserState(ctx context.Context, id string, state store.UserState) error
DeleteUserSessions(ctx context.Context, userID string) error
SetStaffRole(ctx context.Context, id, role string) error
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
RemoveMembership(ctx context.Context, user string, scope store.Scope, id string, at time.Time) (int, error)
AllOrgs(ctx context.Context) ([]store.Org, error)
AllTeams(ctx context.Context) ([]store.TeamPath, error)
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
}
// Mapping says what a group stands for: staff roles by group, and places in
// the tree by group — the same as single sign-on's.
type Mapping struct {
Roles map[string][]string
Memberships map[string]string
}
// Server answers /scim/v2.
type Server struct {
st Store
token string
m Mapping
log *slog.Logger
}
// New builds the server. token is what the provider sends as its bearer.
func New(st Store, token string, m Mapping, log *slog.Logger) *Server {
if log == nil {
log = slog.Default()
}
return &Server{st: st, token: token, m: m, log: log}
}
const (
userSchema = "urn:ietf:params:scim:schemas:core:2.0:User"
groupSchema = "urn:ietf:params:scim:schemas:core:2.0:Group"
listSchema = "urn:ietf:params:scim:api:messages:2.0:ListResponse"
errSchema = "urn:ietf:params:scim:api:messages:2.0:Error"
patchSchema = "urn:ietf:params:scim:api:messages:2.0:PatchOp"
)
// Register mounts the endpoints.
func (s *Server) Register(mux *http.ServeMux) {
h := func(f http.HandlerFunc) http.HandlerFunc { return s.authed(f) }
mux.HandleFunc("GET /scim/v2/ServiceProviderConfig", h(s.config))
mux.HandleFunc("GET /scim/v2/Users", h(s.listUsers))
mux.HandleFunc("POST /scim/v2/Users", h(s.createUser))
mux.HandleFunc("GET /scim/v2/Users/{id}", h(s.getUser))
mux.HandleFunc("PUT /scim/v2/Users/{id}", h(s.replaceUser))
mux.HandleFunc("PATCH /scim/v2/Users/{id}", h(s.patchUser))
mux.HandleFunc("DELETE /scim/v2/Users/{id}", h(s.deleteUser))
mux.HandleFunc("GET /scim/v2/Groups", h(s.listGroups))
mux.HandleFunc("POST /scim/v2/Groups", h(s.createGroup))
mux.HandleFunc("GET /scim/v2/Groups/{id}", h(s.getGroup))
mux.HandleFunc("PATCH /scim/v2/Groups/{id}", h(s.patchGroup))
mux.HandleFunc("PUT /scim/v2/Groups/{id}", h(s.replaceGroup))
mux.HandleFunc("DELETE /scim/v2/Groups/{id}", h(s.deleteGroup))
}
func (s *Server) authed(next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
got, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
if !ok || s.token == "" || subtle.ConstantTimeCompare([]byte(strings.TrimSpace(got)), []byte(s.token)) != 1 {
fail(w, http.StatusUnauthorized, "", "a bearer token is required")
return
}
next(w, r)
}
}
func write(w http.ResponseWriter, code int, v any) {
w.Header().Set("Content-Type", "application/scim+json")
w.WriteHeader(code)
_ = json.NewEncoder(w).Encode(v)
}
func fail(w http.ResponseWriter, code int, scimType, detail string) {
e := map[string]any{"schemas": []string{errSchema}, "status": strconv.Itoa(code), "detail": detail}
if scimType != "" {
e["scimType"] = scimType
}
write(w, code, e)
}
func (s *Server) config(w http.ResponseWriter, _ *http.Request) {
write(w, http.StatusOK, map[string]any{
"schemas": []string{"urn:ietf:params:scim:schemas:core:2.0:ServiceProviderConfig"},
"patch": map[string]bool{"supported": true}, "bulk": map[string]any{"supported": false},
"filter": map[string]any{"supported": true, "maxResults": 1000}, "changePassword": map[string]bool{"supported": false},
"sort": map[string]bool{"supported": false}, "etag": map[string]bool{"supported": false},
"authenticationSchemes": []map[string]string{{"type": "oauthbearertoken", "name": "Bearer token"}},
})
}
// --- users ---
type userIn struct {
UserName string `json:"userName"`
Name struct {
Formatted string `json:"formatted"`
GivenName string `json:"givenName"`
FamilyName string `json:"familyName"`
} `json:"name"`
DisplayName string `json:"displayName"`
Emails []struct {
Value string `json:"value"`
Primary bool `json:"primary"`
} `json:"emails"`
Active *bool `json:"active"`
}
func (u userIn) email() string {
e := u.UserName
if !strings.Contains(e, "@") {
for _, m := range u.Emails {
if m.Primary || e == "" || !strings.Contains(e, "@") {
e = m.Value
}
}
}
return store.NormalizeEmail(e)
}
func (u userIn) name() string {
switch {
case u.Name.Formatted != "":
return u.Name.Formatted
case u.DisplayName != "":
return u.DisplayName
case u.Name.GivenName != "" || u.Name.FamilyName != "":
return strings.TrimSpace(u.Name.GivenName + " " + u.Name.FamilyName)
}
return ""
}
func userOut(u store.User) map[string]any {
return map[string]any{
"schemas": []string{userSchema}, "id": u.ID, "userName": u.Email,
"name": map[string]string{"formatted": u.Name},
"emails": []map[string]any{{"value": u.Email, "primary": true}},
"active": u.State == store.UserActive,
"meta": map[string]any{"resourceType": "User", "created": u.CreatedAt.UTC().Format(time.RFC3339)},
}
}
var userNameEq = regexp.MustCompile(`(?i)^\s*userName\s+eq\s+"([^"]*)"\s*$`)
func page(r *http.Request, n int) (int, int) {
start, _ := strconv.Atoi(r.URL.Query().Get("startIndex"))
if start < 1 {
start = 1
}
count, err := strconv.Atoi(r.URL.Query().Get("count"))
if err != nil || count < 0 {
count = 100
}
if start-1 > n {
start = n + 1
}
return start, min(count, n-(start-1))
}
func (s *Server) listUsers(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
var users []store.User
if f := r.URL.Query().Get("filter"); f != "" {
m := userNameEq.FindStringSubmatch(f)
if m == nil {
fail(w, http.StatusBadRequest, "invalidFilter", `only userName eq "…" is supported`)
return
}
u, err := s.st.UserByEmail(ctx, store.NormalizeEmail(m[1]))
if err != nil && !errors.Is(err, store.ErrNotFound) {
fail(w, http.StatusInternalServerError, "", err.Error())
return
}
if err == nil {
users = []store.User{u}
}
} else {
all, err := s.st.Users(ctx, "")
if err != nil {
fail(w, http.StatusInternalServerError, "", err.Error())
return
}
users = all
}
start, count := page(r, len(users))
res := []map[string]any{}
for _, u := range users[start-1 : start-1+count] {
res = append(res, userOut(u))
}
write(w, http.StatusOK, map[string]any{"schemas": []string{listSchema}, "totalResults": len(users),
"startIndex": start, "itemsPerPage": len(res), "Resources": res})
}
func (s *Server) getUser(w http.ResponseWriter, r *http.Request) {
u, err := s.st.UserByID(r.Context(), r.PathValue("id"))
if err != nil {
fail(w, http.StatusNotFound, "", "no such user")
return
}
write(w, http.StatusOK, userOut(u))
}
func decode(r *http.Request, v any) error {
return json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(v)
}
func newID() string {
var b [10]byte
_, _ = rand.Read(b[:])
return "usr-" + strings.ToLower(base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(b[:]))
}
func (s *Server) createUser(w http.ResponseWriter, r *http.Request) {
var in userIn
if err := decode(r, &in); err != nil {
fail(w, http.StatusBadRequest, "invalidSyntax", err.Error())
return
}
email := in.email()
if !strings.Contains(email, "@") {
fail(w, http.StatusBadRequest, "invalidValue", "userName or an email must be an email address")
return
}
if _, err := s.st.UserByEmail(r.Context(), email); err == nil {
fail(w, http.StatusConflict, "uniqueness", email+" already has an account")
return
}
u := store.User{ID: newID(), Email: email, Name: or(in.name(), email),
// No password works: the provider signs this account in.
PasswordHash: "!scim", State: store.UserActive, CreatedAt: time.Now()}
if in.Active != nil && !*in.Active {
u.State = store.UserSuspended
}
if err := s.st.CreateUser(r.Context(), u); err != nil {
fail(w, http.StatusInternalServerError, "", err.Error())
return
}
s.log.Info("an account was provisioned by the identity provider", "email", email)
write(w, http.StatusCreated, userOut(u))
}
// setActive suspends or reinstates; a suspended account's sessions close.
func (s *Server) setActive(ctx context.Context, u *store.User, active bool) error {
want := store.UserActive
if !active {
want = store.UserSuspended
}
if u.State == want {
return nil
}
if err := s.st.SetUserState(ctx, u.ID, want); err != nil {
return err
}
if !active {
if err := s.st.DeleteUserSessions(ctx, u.ID); err != nil {
return err
}
}
u.State = want
s.log.Info("the identity provider changed an account", "email", u.Email, "state", want)
return nil
}
func (s *Server) replaceUser(w http.ResponseWriter, r *http.Request) {
u, err := s.st.UserByID(r.Context(), r.PathValue("id"))
if err != nil {
fail(w, http.StatusNotFound, "", "no such user")
return
}
var in userIn
if err := decode(r, &in); err != nil {
fail(w, http.StatusBadRequest, "invalidSyntax", err.Error())
return
}
if in.Active != nil {
if err := s.setActive(r.Context(), &u, *in.Active); err != nil {
fail(w, http.StatusInternalServerError, "", err.Error())
return
}
}
write(w, http.StatusOK, userOut(u))
}
type patchOp struct {
Op string `json:"op"`
Path string `json:"path"`
Value json.RawMessage `json:"value"`
}
func (s *Server) patchUser(w http.ResponseWriter, r *http.Request) {
u, err := s.st.UserByID(r.Context(), r.PathValue("id"))
if err != nil {
fail(w, http.StatusNotFound, "", "no such user")
return
}
var in struct{ Operations []patchOp }
if err := decode(r, &in); err != nil {
fail(w, http.StatusBadRequest, "invalidSyntax", err.Error())
return
}
for _, op := range in.Operations {
if active, ok := activeOf(op); ok {
if err := s.setActive(r.Context(), &u, active); err != nil {
fail(w, http.StatusInternalServerError, "", err.Error())
return
}
}
}
write(w, http.StatusOK, userOut(u))
}
// activeOf reads "active" from an operation, written either way providers
// write it: {path: active, value: false} or {value: {active: false}}, the
// value a bool or the string "False".
func activeOf(op patchOp) (bool, bool) {
if !strings.EqualFold(op.Op, "replace") && !strings.EqualFold(op.Op, "add") {
return false, false
}
raw := op.Value
if !strings.EqualFold(op.Path, "active") {
var v map[string]json.RawMessage
if json.Unmarshal(op.Value, &v) != nil {
return false, false
}
var ok bool
if raw, ok = v["active"]; !ok {
return false, false
}
}
var b bool
if json.Unmarshal(raw, &b) == nil {
return b, true
}
var str string
if json.Unmarshal(raw, &str) == nil {
if v, err := strconv.ParseBool(str); err == nil {
return v, true
}
}
return false, false
}
func (s *Server) deleteUser(w http.ResponseWriter, r *http.Request) {
u, err := s.st.UserByID(r.Context(), r.PathValue("id"))
if err != nil {
fail(w, http.StatusNotFound, "", "no such user")
return
}
if err := s.setActive(r.Context(), &u, false); err != nil {
fail(w, http.StatusInternalServerError, "", err.Error())
return
}
w.WriteHeader(http.StatusNoContent)
}
// --- groups ---
// groups are the names the mapping knows, which are the only groups that
// mean anything here.
func (s *Server) groups() []string {
seen := map[string]bool{}
for _, gs := range s.m.Roles {
for _, g := range gs {
seen[g] = true
}
}
for g := range s.m.Memberships {
seen[g] = true
}
out := make([]string, 0, len(seen))
for g := range seen {
out = append(out, g)
}
sort.Strings(out)
return out
}
func groupOut(name string) map[string]any {
return map[string]any{"schemas": []string{groupSchema}, "id": name, "displayName": name,
"meta": map[string]string{"resourceType": "Group"}}
}
func (s *Server) listGroups(w http.ResponseWriter, r *http.Request) {
names := s.groups()
if f := r.URL.Query().Get("filter"); f != "" {
m := regexp.MustCompile(`(?i)^\s*displayName\s+eq\s+"([^"]*)"\s*$`).FindStringSubmatch(f)
if m == nil {
fail(w, http.StatusBadRequest, "invalidFilter", `only displayName eq "…" is supported`)
return
}
names = slices.DeleteFunc(names, func(n string) bool { return n != m[1] })
}
res := []map[string]any{}
for _, n := range names {
res = append(res, groupOut(n))
}
write(w, http.StatusOK, map[string]any{"schemas": []string{listSchema}, "totalResults": len(res),
"startIndex": 1, "itemsPerPage": len(res), "Resources": res})
}
func (s *Server) getGroup(w http.ResponseWriter, r *http.Request) {
if !slices.Contains(s.groups(), r.PathValue("id")) {
fail(w, http.StatusNotFound, "", "no such group: map it under auth.oidc roles or memberships")
return
}
write(w, http.StatusOK, groupOut(r.PathValue("id")))
}
type groupIn struct {
DisplayName string `json:"displayName"`
Members []struct {
Value string `json:"value"`
} `json:"members"`
}
// createGroup answers a provider pushing a group: one the mapping knows is
// followed from now on, its members joined; any other is acknowledged and
// means nothing here.
func (s *Server) createGroup(w http.ResponseWriter, r *http.Request) {
var in groupIn
if err := decode(r, &in); err != nil || in.DisplayName == "" {
fail(w, http.StatusBadRequest, "invalidSyntax", "a group needs a displayName")
return
}
for _, m := range in.Members {
s.member(r.Context(), in.DisplayName, m.Value, true)
}
write(w, http.StatusCreated, groupOut(in.DisplayName))
}
func (s *Server) replaceGroup(w http.ResponseWriter, r *http.Request) {
var in groupIn
if err := decode(r, &in); err != nil {
fail(w, http.StatusBadRequest, "invalidSyntax", err.Error())
return
}
for _, m := range in.Members {
s.member(r.Context(), r.PathValue("id"), m.Value, true)
}
write(w, http.StatusOK, groupOut(r.PathValue("id")))
}
func (s *Server) patchGroup(w http.ResponseWriter, r *http.Request) {
name := r.PathValue("id")
var in struct{ Operations []patchOp }
if err := decode(r, &in); err != nil {
fail(w, http.StatusBadRequest, "invalidSyntax", err.Error())
return
}
for _, op := range in.Operations {
add := strings.EqualFold(op.Op, "add") || strings.EqualFold(op.Op, "replace")
remove := strings.EqualFold(op.Op, "remove")
var members []struct {
Value string `json:"value"`
}
_ = json.Unmarshal(op.Value, &members)
// remove with a path: members[value eq "id"]
if remove && len(members) == 0 {
if m := regexp.MustCompile(`value eq "([^"]+)"`).FindStringSubmatch(op.Path); m != nil {
members = append(members, struct {
Value string `json:"value"`
}{m[1]})
}
}
if !strings.HasPrefix(strings.ToLower(op.Path), "members") && op.Path != "" {
continue
}
for _, m := range members {
if add || remove {
s.member(r.Context(), name, m.Value, add)
}
}
}
w.WriteHeader(http.StatusNoContent)
}
func (s *Server) deleteGroup(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// member applies a person joining or leaving a group: the staff role it
// gives, and the place in the tree.
func (s *Server) member(ctx context.Context, group, userID string, joined bool) {
u, err := s.st.UserByID(ctx, userID)
if err != nil {
s.log.Warn("a group change names a user the gateway does not have", "group", group, "user", userID)
return
}
for _, role := range []string{store.RoleAdmin, store.RoleOperator, store.RoleBilling, store.RoleViewer} {
if !slices.Contains(s.m.Roles[role], group) {
continue
}
switch {
case joined && roleRank(role) > roleRank(u.StaffRole()):
_ = s.st.SetStaffRole(ctx, u.ID, role)
case !joined && u.StaffRole() == role:
_ = s.st.SetStaffRole(ctx, u.ID, "")
}
}
where, ok := s.m.Memberships[group]
if !ok {
return
}
scope, id, err := s.place(ctx, where)
if err != nil {
s.log.Warn("a group maps to a place the tree does not have", "group", group, "place", where, "err", err)
return
}
if joined {
err = s.st.AddMembership(ctx, u.ID, scope, id, store.RoleMember, "scim", time.Now())
} else {
_, err = s.st.RemoveMembership(ctx, u.ID, scope, id, time.Now())
}
if err != nil {
s.log.Warn("a group's membership could not be applied", "group", group, "user", u.Email, "err", err)
}
}
func roleRank(r string) int {
return map[string]int{store.RoleViewer: 1, store.RoleBilling: 2, store.RoleOperator: 3, store.RoleAdmin: 4}[r]
}
func (s *Server) place(ctx context.Context, where string) (store.Scope, string, error) {
kind, path, _ := strings.Cut(where, ":")
// The store writes paths "Acme / ml / train"; a mapping writes them
// either way.
norm := func(s string) string {
parts := strings.Split(s, "/")
for i := range parts {
parts[i] = strings.TrimSpace(parts[i])
}
return strings.Join(parts, "/")
}
same := func(a string) bool { return strings.EqualFold(norm(a), norm(path)) }
switch kind {
case "org":
orgs, err := s.st.AllOrgs(ctx)
for _, o := range orgs {
if same(o.Name) {
return store.ScopeOrg, o.ID, nil
}
}
return "", "", errors.Join(err, fmt.Errorf("no organization %s", path))
case "team":
teams, err := s.st.AllTeams(ctx)
for _, t := range teams {
if same(t.OrgName + "/" + t.Name) {
return store.ScopeTeam, t.ID, nil
}
}
return "", "", errors.Join(err, fmt.Errorf("no team %s", path))
default:
projects, err := s.st.AllProjects(ctx)
for _, p := range projects {
if same(p.Path) {
return store.ScopeProject, p.ID, nil
}
}
return "", "", errors.Join(err, fmt.Errorf("no project %s", path))
}
}
func or(s, def string) string {
if s == "" {
return def
}
return s
}
internal/scim/scim_test.go
+157
View File
@@ -0,0 +1,157 @@
package scim
import (
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
type mem struct {
users map[string]store.User
sessions map[string]bool
roles map[string]string
members map[string]bool
}
func newMem() *mem {
return &mem{users: map[string]store.User{}, sessions: map[string]bool{}, roles: map[string]string{}, members: map[string]bool{}}
}
func (m *mem) Users(context.Context, store.UserState) ([]store.User, error) {
var out []store.User
for _, u := range m.users {
out = append(out, u)
}
return out, nil
}
func (m *mem) UserByID(_ context.Context, id string) (store.User, error) {
if u, ok := m.users[id]; ok {
u.Role = m.roles[id]
return u, nil
}
return store.User{}, store.ErrNotFound
}
func (m *mem) UserByEmail(_ context.Context, e string) (store.User, error) {
for _, u := range m.users {
if u.Email == e {
return u, nil
}
}
return store.User{}, store.ErrNotFound
}
func (m *mem) CreateUser(_ context.Context, u store.User) error { m.users[u.ID] = u; return nil }
func (m *mem) SetUserState(_ context.Context, id string, s store.UserState) error {
u := m.users[id]
u.State = s
m.users[id] = u
return nil
}
func (m *mem) DeleteUserSessions(_ context.Context, id string) error {
delete(m.sessions, id)
return nil
}
func (m *mem) SetStaffRole(_ context.Context, id, role string) error { m.roles[id] = role; return nil }
func (m *mem) AddMembership(_ context.Context, u string, sc store.Scope, id, _, _ string, _ time.Time) error {
m.members[u+"@"+string(sc)+":"+id] = true
return nil
}
func (m *mem) RemoveMembership(_ context.Context, u string, sc store.Scope, id string, _ time.Time) (int, error) {
delete(m.members, u+"@"+string(sc)+":"+id)
return 1, nil
}
func (m *mem) AllOrgs(context.Context) ([]store.Org, error) { return nil, nil }
func (m *mem) AllTeams(context.Context) ([]store.TeamPath, error) {
return []store.TeamPath{{ID: "t1", Name: "ml", OrgName: "Acme"}}, nil
}
func (m *mem) AllProjects(context.Context) ([]store.ProjectPath, error) {
// As the store writes them.
return []store.ProjectPath{{ID: "p1", Path: "Acme / ml / train"}}, nil
}
func server(t *testing.T) (*mem, http.Handler) {
st := newMem()
mux := http.NewServeMux()
New(st, "s3cret", Mapping{Roles: map[string][]string{"operator": {"ops"}},
Memberships: map[string]string{"ml": "team:Acme/ml", "trainers": "project:Acme/ml/train"}}, slog.New(slog.DiscardHandler)).Register(mux)
return st, mux
}
func do(t *testing.T, h http.Handler, method, path, body string) (*httptest.ResponseRecorder, map[string]any) {
t.Helper()
r := httptest.NewRequest(method, path, strings.NewReader(body))
r.Header.Set("Authorization", "Bearer s3cret")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, r)
var out map[string]any
_ = json.Unmarshal(rec.Body.Bytes(), &out)
return rec, out
}
func TestAProviderProvisionsAndSuspendsAccounts(t *testing.T) {
st, h := server(t)
r := httptest.NewRequest("GET", "/scim/v2/Users", nil)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, r)
if rec.Code != 401 {
t.Errorf("no token: %d", rec.Code)
}
rec, u := do(t, h, "POST", "/scim/v2/Users", `{"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],
"userName":"Ann@Example.com","name":{"givenName":"Ann","familyName":"Lee"},"active":true}`)
if rec.Code != 201 || u["userName"] != "ann@example.com" || u["active"] != true {
t.Fatalf("create: %d %v", rec.Code, u)
}
id := u["id"].(string)
if st.users[id].Name != "Ann Lee" || st.users[id].PasswordHash != "!scim" {
t.Errorf("stored %+v", st.users[id])
}
if rec, _ := do(t, h, "POST", "/scim/v2/Users", `{"userName":"ann@example.com"}`); rec.Code != 409 {
t.Errorf("again: %d", rec.Code)
}
if _, l := do(t, h, "GET", `/scim/v2/Users?filter=userName+eq+%22ANN@example.com%22`, ""); l["totalResults"] != float64(1) {
t.Errorf("filter: %v", l)
}
// Entra writes {op: Replace, path: active, value: "False"}; Okta {op:
// replace, value: {active: false}}.
st.sessions[id] = true
do(t, h, "PATCH", "/scim/v2/Users/"+id, `{"Operations":[{"op":"Replace","path":"active","value":"False"}]}`)
if st.users[id].State != store.UserSuspended || st.sessions[id] {
t.Errorf("suspend: %+v sessions %v", st.users[id], st.sessions[id])
}
do(t, h, "PATCH", "/scim/v2/Users/"+id, `{"Operations":[{"op":"replace","value":{"active":true}}]}`)
if st.users[id].State != store.UserActive {
t.Errorf("reinstate: %+v", st.users[id])
}
if rec, _ := do(t, h, "DELETE", "/scim/v2/Users/"+id, ""); rec.Code != 204 || st.users[id].State != store.UserSuspended {
t.Errorf("delete is suspend: %d %+v", rec.Code, st.users[id])
}
// Groups the mapping knows give roles and places.
do(t, h, "PATCH", "/scim/v2/Users/"+id, `{"Operations":[{"op":"replace","value":{"active":true}}]}`)
do(t, h, "PATCH", "/scim/v2/Groups/ops", `{"Operations":[{"op":"add","path":"members","value":[{"value":"`+id+`"}]}]}`)
do(t, h, "PATCH", "/scim/v2/Groups/ml", `{"Operations":[{"op":"add","path":"members","value":[{"value":"`+id+`"}]}]}`)
if st.roles[id] != "operator" || !st.members[id+"@team:t1"] {
t.Errorf("roles %v members %v", st.roles, st.members)
}
do(t, h, "PATCH", "/scim/v2/Groups/ml", `{"Operations":[{"op":"remove","path":"members[value eq \"`+id+`\"]"}]}`)
do(t, h, "PATCH", "/scim/v2/Groups/ops", `{"Operations":[{"op":"remove","path":"members","value":[{"value":"`+id+`"}]}]}`)
if st.roles[id] != "" || st.members[id+"@team:t1"] {
t.Errorf("after leaving: roles %v members %v", st.roles, st.members)
}
do(t, h, "PATCH", "/scim/v2/Groups/trainers", `{"Operations":[{"op":"add","path":"members","value":[{"value":"`+id+`"}]}]}`)
if !st.members[id+"@project:p1"] {
t.Errorf("a project written without spaces: %v", st.members)
}
if _, l := do(t, h, "GET", "/scim/v2/Groups", ""); l["totalResults"] != float64(3) {
t.Errorf("groups: %v", l)
}
}
internal/store/audit.go
+72
View File
@@ -0,0 +1,72 @@
package store
import (
"context"
"database/sql"
"strings"
"time"
)
// AuditEvent is one change: who made it, through what, to what, and how it
// ended. Detail never holds a secret; the writer takes them out.
type AuditEvent struct {
ID int64 `json:"id"`
At time.Time `json:"at"`
Actor string `json:"actor"`
Source string `json:"source"` // api, web
Action string `json:"action"` // "POST /api/v1/models", "POST /admin/users/{id}/suspend"
Target string `json:"target,omitempty"`
Detail string `json:"detail,omitempty"`
Status int `json:"status"`
}
// WriteAudit records one change.
func (s *DB) WriteAudit(ctx context.Context, e AuditEvent) error {
if e.At.IsZero() {
e.At = time.Now()
}
_, err := s.db.ExecContext(ctx, `INSERT INTO audit_events (ts, actor, source, action, target, detail, status)
VALUES (?, ?, ?, ?, ?, ?, ?)`, millis(e.At), e.Actor, e.Source, e.Action, e.Target, e.Detail, e.Status)
return err
}
// AuditFilter narrows a listing; empty fields match everything.
type AuditFilter struct {
Since, Until time.Time
Actor string
// Contains matches the action or the target.
Contains string
Limit int
}
// AuditEvents lists changes, the newest first.
func (s *DB) AuditEvents(ctx context.Context, f AuditFilter) ([]AuditEvent, error) {
q := `SELECT id, ts, actor, source, action, target, detail, status FROM audit_events WHERE ts >= ?`
args := []any{millis(f.Since)}
if !f.Until.IsZero() {
q += ` AND ts < ?`
args = append(args, millis(f.Until))
}
if f.Actor != "" {
q += ` AND actor = ?`
args = append(args, f.Actor)
}
if f.Contains != "" {
q += ` AND (LOWER(action) LIKE ? OR LOWER(target) LIKE ?)`
like := "%" + strings.ToLower(f.Contains) + "%"
args = append(args, like, like)
}
limit := f.Limit
if limit <= 0 || limit > 10000 {
limit = 200
}
q += ` ORDER BY ts DESC, id DESC LIMIT ?`
args = append(args, limit)
return queryAll(ctx, s.db, q, args, func(rows *sql.Rows) (AuditEvent, error) {
var e AuditEvent
var ts int64
err := rows.Scan(&e.ID, &ts, &e.Actor, &e.Source, &e.Action, &e.Target, &e.Detail, &e.Status)
e.At = fromMillis(ts)
return e, err
})
}
internal/store/audit_test.go
+55
View File
@@ -0,0 +1,55 @@
package store
import (
"testing"
"time"
)
func TestAuditEventsAreKeptAndFound(t *testing.T) {
db := capacityStore(t)
ctx := t.Context()
now := time.Now()
for i, e := range []AuditEvent{
{At: now.Add(-2 * time.Hour), Actor: "ilker", Source: "api", Action: "POST /api/v1/models", Target: "m1", Status: 200},
{At: now.Add(-time.Hour), Actor: "ann@example.com", Source: "web", Action: "POST /admin/users/u1/suspend", Target: "bob@example.com", Status: 200},
{At: now, Actor: "ilker", Source: "api", Action: "DELETE /api/v1/nodes/n1", Status: 409},
} {
if err := db.WriteAudit(ctx, e); err != nil {
t.Fatal(i, err)
}
}
all, err := db.AuditEvents(ctx, AuditFilter{Since: now.Add(-24 * time.Hour)})
if err != nil || len(all) != 3 || all[0].Action != "DELETE /api/v1/nodes/n1" {
t.Fatalf("%+v %v", all, err)
}
if got, _ := db.AuditEvents(ctx, AuditFilter{Since: now.Add(-24 * time.Hour), Actor: "ilker"}); len(got) != 2 {
t.Errorf("by actor: %d", len(got))
}
if got, _ := db.AuditEvents(ctx, AuditFilter{Since: now.Add(-24 * time.Hour), Contains: "BOB@"}); len(got) != 1 {
t.Errorf("by target: %d", len(got))
}
if got, _ := db.AuditEvents(ctx, AuditFilter{Since: now.Add(-90 * time.Minute)}); len(got) != 2 {
t.Errorf("since: %d", len(got))
}
}
func TestAStaffRoleIsGivenAndTakenAway(t *testing.T) {
db := capacityStore(t)
ctx := t.Context()
if err := db.CreateUser(ctx, User{ID: "u1", Email: "v@example.com", Name: "v", PasswordHash: "x",
State: UserActive, CreatedAt: time.Now()}); err != nil {
t.Fatal(err)
}
for _, role := range []string{RoleViewer, RoleOperator, RoleBilling, RoleAdmin, ""} {
if err := db.SetStaffRole(ctx, "u1", role); err != nil {
t.Fatal(role, err)
}
u, err := db.UserByID(ctx, "u1")
if err != nil || u.StaffRole() != role || u.Staff() != (role != "") || u.IsAdmin != (role == RoleAdmin) {
t.Errorf("%q: %+v %v", role, u, err)
}
}
if db.SetStaffRole(ctx, "u1", "god") == nil {
t.Error("an unknown role was taken")
}
}
internal/store/copy.go
+4 -1
View File
@@ -55,6 +55,9 @@ var copyOrder = []string{
// Or a removed node's certificate would work again in the new store.
"revoked_nodes",
"guard_attachments",
// Who changed what is not lost by moving the store.
"audit_events",
"key_ttl",
}
// notCopied are tables deliberately left behind.
@@ -227,7 +230,7 @@ func (s *DB) resetIdentities(ctx context.Context) error {
return nil
}
// Only the tables with a generated id. SQLite's rowid needs nothing.
for _, t := range []string{"memory_notes", "usage"} {
for _, t := range []string{"memory_notes", "usage", "audit_events"} {
var has bool
if err := s.db.QueryRowContext(ctx,
`SELECT EXISTS (SELECT 1 FROM information_schema.columns
internal/store/export.go
+152
View File
@@ -0,0 +1,152 @@
package store
import (
"context"
"database/sql"
"fmt"
"slices"
"strings"
"time"
)
// ExportRow is one line of a usage export: a period, the place, person, key
// and model it is about — those asked for, the rest empty — and what was used.
type ExportRow struct {
Period time.Time
Org string
Team string
Project string
User string
Key string
Model string
Requests int64
Errors int64
Limited int64
Prompt int64
Completion int64
CostMicros int64
}
// ExportDims are what an export can be broken down by.
var ExportDims = []string{"org", "team", "project", "user", "key", "model"}
// UsageExport totals usage between two moments, by period — hour, day or
// month, in UTC — and by the dimensions asked for. It reads the hourly and
// daily sums, so request records past their retention are still counted.
func (s *DB) UsageExport(ctx context.Context, from, to time.Time, period string, by []string) ([]ExportRow, error) {
for _, d := range by {
if !slices.Contains(ExportDims, d) {
return nil, fmt.Errorf("%q is not one of %s", d, strings.Join(ExportDims, ", "))
}
}
table := "usage_1d"
switch period {
case "hour":
table = "usage_1h"
case "day", "month":
default:
return nil, fmt.Errorf("period %q is not hour, day or month", period)
}
type raw struct {
bucket int64
key, user, project, model string
requests, errors, limited, prompt, completion, cost int64
}
//nolint:gosec // table is one of two literals above
rows, err := queryAll(ctx, s.db, `SELECT bucket, key_id, user_id, project_id, served_model,
SUM(requests), SUM(errors), SUM(limited), SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros)
FROM `+table+` WHERE bucket >= ? AND bucket < ?
GROUP BY bucket, key_id, user_id, project_id, served_model`,
[]any{millis(from), millis(to)}, func(r *sql.Rows) (raw, error) {
var x raw
err := r.Scan(&x.bucket, &x.key, &x.user, &x.project, &x.model,
&x.requests, &x.errors, &x.limited, &x.prompt, &x.completion, &x.cost)
return x, err
})
if err != nil {
return nil, err
}
// Names, not ids: an export is read by people and spreadsheets.
type place struct{ org, team, project string }
places := map[string]place{"": {project: "personal"}}
tree, err := s.Tree(ctx)
if err != nil {
return nil, err
}
for _, o := range tree {
for _, t := range o.Teams {
for _, p := range t.Projects {
places[p.ID] = place{o.Name, t.Name, p.Name}
}
}
}
emails := map[string]string{}
users, err := s.Users(ctx, "")
if err != nil {
return nil, err
}
for _, u := range users {
emails[u.ID] = u.Email
}
has := func(d string) bool { return slices.Contains(by, d) }
totals := map[ExportRow]*ExportRow{}
var order []ExportRow
for _, x := range rows {
at := fromMillis(x.bucket).UTC()
if period == "month" {
at = time.Date(at.Year(), at.Month(), 1, 0, 0, 0, 0, time.UTC)
}
pl := places[x.project]
k := ExportRow{Period: at}
if has("org") {
k.Org = pl.org
}
if has("team") {
k.Team = pl.team
}
if has("project") {
k.Project = pl.project
if pl.project == "" {
k.Project = x.project
}
}
if has("user") {
k.User = emails[x.user]
if k.User == "" {
k.User = x.user
}
}
if has("key") {
k.Key = x.key
}
if has("model") {
k.Model = x.model
}
t, ok := totals[k]
if !ok {
row := k
t = &row
totals[k] = t
order = append(order, k)
}
t.Requests += x.requests
t.Errors += x.errors
t.Limited += x.limited
t.Prompt += x.prompt
t.Completion += x.completion
t.CostMicros += x.cost
}
out := make([]ExportRow, 0, len(order))
for _, k := range order {
out = append(out, *totals[k])
}
slices.SortFunc(out, func(a, b ExportRow) int {
if c := a.Period.Compare(b.Period); c != 0 {
return c
}
return strings.Compare(a.Org+a.Team+a.Project+a.User+a.Key+a.Model, b.Org+b.Team+b.Project+b.User+b.Key+b.Model)
})
return out, nil
}
internal/store/export_test.go
+48
View File
@@ -0,0 +1,48 @@
package store
import (
"testing"
"time"
)
func TestUsageIsExportedByPeriodAndWhatWasAskedFor(t *testing.T) {
db := capacityStore(t)
ctx := t.Context()
org, _ := db.CreateOrg(ctx, "acme")
team, _ := db.CreateTeam(ctx, org, "ml")
project, _ := db.CreateProject(ctx, team, "train")
day1 := time.Date(2026, 9, 30, 10, 0, 0, 0, time.UTC)
day2 := time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC)
var rows []Usage
for _, r := range []struct {
at time.Time
model string
p int64
}{{day1, "a", 10}, {day1, "b", 20}, {day2, "a", 5}, {day2.Add(time.Hour), "a", 5}} {
rows = append(rows, Usage{At: r.at, KeyID: "k", ProjectID: project, Model: r.model, ServedModel: r.model,
PromptTokens: r.p, CompletionTokens: 1, CostMicros: 100, Status: 200})
}
if err := db.WriteUsage(ctx, rows); err != nil {
t.Fatal(err)
}
from, to := day1.Add(-24*time.Hour), day2.Add(48*time.Hour)
byDay, err := db.UsageExport(ctx, from, to, "day", []string{"project", "model"})
if err != nil || len(byDay) != 3 {
t.Fatalf("%+v %v", byDay, err)
}
last := byDay[2]
if last.Project != "train" || last.Model != "a" || last.Prompt != 10 || last.Requests != 2 || !last.Period.Equal(time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)) {
t.Errorf("%+v", last)
}
byMonth, _ := db.UsageExport(ctx, from, to, "month", []string{"org"})
if len(byMonth) != 2 || byMonth[0].Org != "acme" || byMonth[0].Prompt != 30 || byMonth[1].CostMicros != 200 {
t.Errorf("by month: %+v", byMonth)
}
if _, err := db.UsageExport(ctx, from, to, "week", nil); err == nil {
t.Error("a week was taken")
}
if _, err := db.UsageExport(ctx, from, to, "day", []string{"planet"}); err == nil {
t.Error("an unknown dimension was taken")
}
}
internal/store/keyttl.go
+123
View File
@@ -0,0 +1,123 @@
package store
import (
"context"
"database/sql"
"errors"
"fmt"
"time"
)
// ScopeGateway is the whole gateway, for settings that apply everywhere.
const ScopeGateway Scope = "gateway"
// ErrKeyTTL is a key asked to live longer than a scope allows.
var ErrKeyTTL = errors.New("a key may not live that long here")
// SetMaxKeyTTL sets how long a key may live at most in a scope; zero takes
// the limit away.
func (s *DB) SetMaxKeyTTL(ctx context.Context, scope Scope, id string, max time.Duration) error {
if max <= 0 {
_, err := s.db.ExecContext(ctx, `DELETE FROM key_ttl WHERE scope = ? AND scope_id = ?`, string(scope), id)
return err
}
_, err := s.db.ExecContext(ctx, `INSERT INTO key_ttl (scope, scope_id, max_seconds) VALUES (?, ?, ?)
ON CONFLICT (scope, scope_id) DO UPDATE SET max_seconds = excluded.max_seconds`,
string(scope), id, int64(max/time.Second))
return err
}
// KeyTTLs lists every limit, "scope:id" to the longest a key may live.
func (s *DB) KeyTTLs(ctx context.Context) (map[string]time.Duration, error) {
type row struct {
k string
d time.Duration
}
rows, err := queryAll(ctx, s.db, `SELECT scope, scope_id, max_seconds FROM key_ttl`, nil,
func(r *sql.Rows) (row, error) {
var sc, id string
var n int64
err := r.Scan(&sc, &id, &n)
return row{sc + ":" + id, time.Duration(n) * time.Second}, err
})
out := map[string]time.Duration{}
for _, r := range rows {
out[r.k] = r.d
}
return out, err
}
// MaxKeyTTL is the longest a key in a project may live, and the scope that
// says so: the shortest of the gateway's, and the project's, its team's and
// its organization's. Zero when nothing limits it. A personal key (no
// project) has only the gateway's.
func (s *DB) MaxKeyTTL(ctx context.Context, projectID string) (time.Duration, string, error) {
limits, err := s.KeyTTLs(ctx)
if err != nil || len(limits) == 0 {
return 0, "", err
}
scopes := []string{string(ScopeGateway) + ":"}
if projectID != "" {
parents, err := s.ScopeParents(ctx)
if err != nil {
return 0, "", err
}
scopes = append(scopes, "project:"+projectID)
scopes = append(scopes, parents["project:"+projectID]...)
}
var best time.Duration
by := ""
for _, sc := range scopes {
if d, ok := limits[sc]; ok && (best == 0 || d < best) {
best, by = d, sc
}
}
return best, by, nil
}
// keyTTL gives a key that asks for no expiry the longest its scope allows,
// and refuses one that asks to outlive it.
func (s *DB) keyTTL(ctx context.Context, k Key) (Key, error) {
max, by, err := s.MaxKeyTTL(ctx, k.ProjectID)
if err != nil || max == 0 {
return k, err
}
created := k.CreatedAt
if created.IsZero() {
created = time.Now()
}
if k.ExpiresAt.IsZero() {
k.ExpiresAt = created.Add(max)
return k, nil
}
if k.ExpiresAt.Sub(created) > max+time.Minute {
return k, fmt.Errorf("%w: %s allows %s at most", ErrKeyTTL, by, max)
}
return k, nil
}
// KeyByID finds a key by its id.
func (s *DB) KeyByID(ctx context.Context, id string) (Key, error) {
row := s.db.QueryRowContext(ctx, `SELECT `+keyColumns+` FROM api_keys WHERE id = ?`, id)
k, err := scanKey(row)
if errors.Is(err, sql.ErrNoRows) {
return Key{}, ErrNotFound
}
return k, err
}
// ExpireKeyBy has a key stop working at a moment, unless it stops sooner
// already: what rotating it does to the old one.
func (s *DB) ExpireKeyBy(ctx context.Context, id string, at time.Time) error {
res, err := s.db.ExecContext(ctx, `UPDATE api_keys SET expires_at = ?
WHERE id = ? AND revoked_at IS NULL AND (expires_at IS NULL OR expires_at > ?)`, millis(at), id, millis(at))
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
if _, err := s.KeyByID(ctx, id); err != nil {
return err
}
}
return nil
}
internal/store/keyttl_test.go
+52
View File
@@ -0,0 +1,52 @@
package store
import (
"errors"
"testing"
"time"
)
func TestAKeyIsMadeWithinTheShortestLifeItsScopesAllow(t *testing.T) {
db := capacityStore(t)
ctx := t.Context()
org, _ := db.CreateOrg(ctx, "acme")
team, _ := db.CreateTeam(ctx, org, "ml")
project, _ := db.CreateProject(ctx, team, "train")
key := func(id string, life time.Duration) Key {
k := Key{ID: id, Hash: []byte(id), Prefix: id, Name: id, ProjectID: project, CreatedAt: time.Now()}
if life > 0 {
k.ExpiresAt = k.CreatedAt.Add(life)
}
return k
}
if err := db.CreateKey(ctx, key("k0", 0)); err != nil {
t.Fatalf("nothing limits it: %v", err)
}
_ = db.SetMaxKeyTTL(ctx, ScopeGateway, "", 90*24*time.Hour)
_ = db.SetMaxKeyTTL(ctx, ScopeTeam, team, 30*24*time.Hour)
if max, by, _ := db.MaxKeyTTL(ctx, project); max != 30*24*time.Hour || by != "team:"+team {
t.Errorf("max %s by %s", max, by)
}
// One that asks for no expiry is given the longest allowed.
if err := db.CreateKey(ctx, key("k1", 0)); err != nil {
t.Errorf("one that asks for no expiry: %v", err)
}
if k, err := db.KeyByHash(ctx, []byte("k1")); err != nil || k.ExpiresAt.Sub(k.CreatedAt) < 29*24*time.Hour ||
k.ExpiresAt.Sub(k.CreatedAt) > 31*24*time.Hour {
t.Errorf("given %v %v", k.ExpiresAt.Sub(k.CreatedAt), err)
}
if err := db.CreateKey(ctx, key("k2", 60*24*time.Hour)); !errors.Is(err, ErrKeyTTL) {
t.Errorf("one past the team's: %v", err)
}
if err := db.CreateKey(ctx, key("k3", 7*24*time.Hour)); err != nil {
t.Errorf("one within: %v", err)
}
// A personal key has only the gateway's.
if max, _, _ := db.MaxKeyTTL(ctx, ""); max != 90*24*time.Hour {
t.Errorf("personal %s", max)
}
_ = db.SetMaxKeyTTL(ctx, ScopeTeam, team, 0)
if max, _, _ := db.MaxKeyTTL(ctx, project); max != 90*24*time.Hour {
t.Errorf("after the team's is taken away: %s", max)
}
}
internal/store/migrations/postgres/0020_audit.sql
+15
View File
@@ -0,0 +1,15 @@
-- Ported from the SQLite migration of the same number.
-- Who changed what, from where: every change through the admin API and the
-- administration pages. Detail is the request's fields with anything secret
-- taken out; status is how it ended.
CREATE TABLE audit_events (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
ts BIGINT NOT NULL,
actor TEXT NOT NULL,
source TEXT NOT NULL,
action TEXT NOT NULL,
target TEXT NOT NULL DEFAULT '',
detail TEXT NOT NULL DEFAULT '',
status INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX audit_events_ts ON audit_events (ts);
internal/store/migrations/postgres/0021_staff_role.sql
+5
View File
@@ -0,0 +1,5 @@
-- Ported from the SQLite migration of the same number.
-- What a person may do on the administration pages besides administering:
-- viewer (look at everything), operator (and run the cluster: models,
-- mirror, nodes), billing (usage and quotas). Administrators keep is_admin.
ALTER TABLE users ADD COLUMN staff_role TEXT NOT NULL DEFAULT '';
internal/store/migrations/postgres/0022_key_ttl.sql
+10
View File
@@ -0,0 +1,10 @@
-- Ported from the SQLite migration of the same number.
-- How long a key may live, at most: for the whole gateway (scope gateway,
-- written from server.yaml when the server starts) and for an organization,
-- team or project. A key is made within the shortest that applies to it.
CREATE TABLE key_ttl (
scope TEXT NOT NULL,
scope_id TEXT NOT NULL,
max_seconds BIGINT NOT NULL,
PRIMARY KEY (scope, scope_id)
);
internal/store/migrations/sqlite/0020_audit.sql
+14
View File
@@ -0,0 +1,14 @@
-- Who changed what, from where: every change through the admin API and the
-- administration pages. Detail is the request's fields with anything secret
-- taken out; status is how it ended.
CREATE TABLE audit_events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts INTEGER NOT NULL,
actor TEXT NOT NULL,
source TEXT NOT NULL,
action TEXT NOT NULL,
target TEXT NOT NULL DEFAULT '',
detail TEXT NOT NULL DEFAULT '',
status INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX audit_events_ts ON audit_events (ts);
internal/store/migrations/sqlite/0021_staff_role.sql
+4
View File
@@ -0,0 +1,4 @@
-- What a person may do on the administration pages besides administering:
-- viewer (look at everything), operator (and run the cluster: models,
-- mirror, nodes), billing (usage and quotas). Administrators keep is_admin.
ALTER TABLE users ADD COLUMN staff_role TEXT NOT NULL DEFAULT '';
internal/store/migrations/sqlite/0022_key_ttl.sql
+9
View File
@@ -0,0 +1,9 @@
-- How long a key may live, at most: for the whole gateway (scope gateway,
-- written from server.yaml when the server starts) and for an organization,
-- team or project. A key is made within the shortest that applies to it.
CREATE TABLE key_ttl (
scope TEXT NOT NULL,
scope_id TEXT NOT NULL,
max_seconds INTEGER NOT NULL,
PRIMARY KEY (scope, scope_id)
);
internal/store/sessions.go
+2 -2
View File
@@ -58,7 +58,7 @@ func (s *DB) SessionByToken(ctx context.Context, token string) (Session, User, e
row := s.db.QueryRowContext(ctx,
`SELECT s.user_id, s.csrf_hash, s.expires_at, s.last_seen,
u.id, u.email, u.name, u.password_hash, u.state, COALESCE(u.project_id, ''),
u.created_at, u.approved_at, u.approved_by, u.last_login_at, u.is_admin
u.created_at, u.approved_at, u.approved_by, u.last_login_at, u.is_admin, u.staff_role
FROM sessions s JOIN users u ON u.id = s.user_id
WHERE s.token_hash = ?`, HashKey(token))
@@ -72,7 +72,7 @@ func (s *DB) SessionByToken(ctx context.Context, token string) (Session, User, e
if err := row.Scan(&sess.UserID, &sess.CSRFHash, &expires, &lastSeen,
&u.ID, &u.Email, &u.Name, &u.PasswordHash, &state, &u.ProjectID,
&created, &approved, &approvedBy, &lastLogin, &u.IsAdmin); err != nil {
&created, &approved, &approvedBy, &lastLogin, &u.IsAdmin, &u.Role); err != nil {
if errors.Is(err, sql.ErrNoRows) {
return Session{}, User{}, ErrNoSession
}
internal/store/sqlite.go
+5 -1
View File
@@ -168,7 +168,11 @@ func (s *DB) CreateKey(ctx context.Context, k Key) error {
if len(k.Hash) == 0 {
return errors.New("store: a key needs a hash")
}
_, err := s.db.ExecContext(ctx,
k, err := s.keyTTL(ctx, k)
if err != nil {
return err
}
_, err = s.db.ExecContext(ctx,
`INSERT INTO api_keys (`+keyColumns+`) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
k.ID, k.Hash, k.Prefix, k.Name, nullString(k.ProjectID),
nullString(k.Band), nullString(k.RoutingProfile),
internal/store/users.go
+53 -4
View File
@@ -4,6 +4,7 @@ import (
"context"
"database/sql"
"errors"
"fmt"
"strings"
"time"
@@ -42,9 +43,36 @@ type User struct {
ApprovedAt time.Time
ApprovedBy string
LastLoginAt time.Time
// IsAdmin lets this account see the cluster dashboard. It is not a role
// system: one bit, and everything that changes the cluster stays on the CLI.
// IsAdmin lets this account administer: every page under /admin, and
// every change there.
IsAdmin bool
// Role is a lesser part in running the gateway: RoleViewer looks at
// every administration page, RoleOperator also runs the cluster from them
// (models, the mirror, nodes), RoleBilling sees usage and quotas. Empty
// for everybody else.
Role string
}
// Staff roles.
const (
RoleViewer = "viewer"
RoleOperator = "operator"
RoleBilling = "billing"
RoleAdmin = "admin"
)
// StaffRoles are the roles SetStaffRole takes; "" takes them all away.
var StaffRoles = []string{RoleViewer, RoleOperator, RoleBilling, RoleAdmin}
// Staff reports whether a person has any part in running the gateway.
func (u User) Staff() bool { return u.IsAdmin || u.Role != "" }
// StaffRole is the role a person holds, admin included.
func (u User) StaffRole() string {
if u.IsAdmin {
return RoleAdmin
}
return u.Role
}
// CanSignIn reports whether this account may open a session.
@@ -58,7 +86,7 @@ func NormalizeEmail(email string) string {
}
const userColumns = `id, email, name, password_hash, state, COALESCE(project_id, ''),
created_at, approved_at, approved_by, last_login_at, is_admin`
created_at, approved_at, approved_by, last_login_at, is_admin, staff_role`
// CreateUser records a new, unapproved account.
func (s *DB) CreateUser(ctx context.Context, u User) error {
@@ -192,6 +220,27 @@ func (s *DB) SetAdmin(ctx context.Context, id string, admin bool) error {
return nil
}
// SetStaffRole gives a person a part in running the gateway — viewer,
// operator, billing or admin — or, with "", none.
func (s *DB) SetStaffRole(ctx context.Context, id, role string) error {
admin, rest := 0, role
switch role {
case RoleAdmin:
admin, rest = 1, ""
case "", RoleViewer, RoleOperator, RoleBilling:
default:
return fmt.Errorf("role %q is not one of viewer, operator, billing, admin", role)
}
res, err := s.db.ExecContext(ctx, `UPDATE users SET is_admin = ?, staff_role = ? WHERE id = ?`, admin, rest, id)
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return ErrNotFound
}
return nil
}
// TouchLogin records a successful sign-in.
func (s *DB) TouchLogin(ctx context.Context, id string, at time.Time) error {
_, err := s.db.ExecContext(ctx, `UPDATE users SET last_login_at = ? WHERE id = ?`,
@@ -252,7 +301,7 @@ func (s *DB) scanUser(sc scanner) (User, error) {
var approvedBy sql.NullString
if err := sc.Scan(&u.ID, &u.Email, &u.Name, &u.PasswordHash, &state, &u.ProjectID,
&created, &approved, &approvedBy, &lastLogin, &u.IsAdmin); err != nil {
&created, &approved, &approvedBy, &lastLogin, &u.IsAdmin, &u.Role); err != nil {
if errors.Is(err, sql.ErrNoRows) {
return User{}, ErrNotFound
}
internal/web/account.go
+33 -7
View File
@@ -14,6 +14,7 @@ import (
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/account"
"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
"gitea.mynodes.xyz/ilker/llmgw/internal/clientip"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -65,6 +66,10 @@ type AccountPages struct {
hasMirror bool
// hasQuotas is set when /admin/quotas is registered.
hasQuotas bool
// oidc is sign-in through an identity provider; nil when there is none.
oidc *oidcLogin
// audit records every change made through these pages; nil records none.
audit audit.Sink
// hasGuard is set when /admin/guard is registered.
hasGuard bool
// guardFor names the guardrail policies in force for someone, and
@@ -150,10 +155,10 @@ func (p *AccountPages) Register(mux *http.ServeMux) {
mux.HandleFunc("POST /login", rationed(newRation(30, 30), p.login))
mux.HandleFunc("POST /logout", p.logout)
mux.HandleFunc("GET /profile", p.profile)
mux.HandleFunc("POST /profile/keys", p.createKey)
mux.HandleFunc("POST /profile/keys/{id}/revoke", p.revokeKey)
mux.HandleFunc("POST /profile/password", p.changePassword)
mux.HandleFunc("POST /profile/connect", p.connect)
mux.HandleFunc("POST /profile/keys", p.selfAudited(p.createKey))
mux.HandleFunc("POST /profile/keys/{id}/revoke", p.selfAudited(p.revokeKey))
mux.HandleFunc("POST /profile/password", p.selfAudited(p.changePassword))
mux.HandleFunc("POST /profile/connect", p.selfAudited(p.connect))
mux.HandleFunc("GET /setup/{tool}", p.setupScript)
}
@@ -201,17 +206,27 @@ func (p *AccountPages) signup(w http.ResponseWriter, r *http.Request) {
// --- sign in and out ---
// ssoLabel is what the sign-in button names, or "" without one.
func (p *AccountPages) ssoLabel() string {
if p.oidc == nil {
return ""
}
return or(p.oidc.cfg.Label, "single sign-on")
}
type loginData struct {
Email, Error string
SignupOpen bool
Version string
// SSO is the identity provider's label, when sign-in through one is on.
SSO string
// CSRF is the form's token, matched by a cookie.
CSRF string
}
func (p *AccountPages) loginForm(w http.ResponseWriter, r *http.Request) {
noStore(w)
render(w, "login", loginData{SignupOpen: p.accounts.SignupOpen(), Version: p.version, CSRF: setLoginToken(w, r)})
render(w, "login", loginData{SSO: p.ssoLabel(), SignupOpen: p.accounts.SignupOpen(), Version: p.version, CSRF: setLoginToken(w, r)})
}
func (p *AccountPages) login(w http.ResponseWriter, r *http.Request) {
@@ -220,7 +235,7 @@ func (p *AccountPages) login(w http.ResponseWriter, r *http.Request) {
if !loginTokenOK(r) {
// Not this site's form: another one posting a visitor into an
// account of its choosing, or a form left open past its hour.
renderStatus(w, http.StatusBadRequest, "login", loginData{Email: email, SignupOpen: p.accounts.SignupOpen(), Version: p.version,
renderStatus(w, http.StatusBadRequest, "login", loginData{SSO: p.ssoLabel(), Email: email, SignupOpen: p.accounts.SignupOpen(), Version: p.version,
Error: "the form had expired; sign in again", CSRF: setLoginToken(w, r)})
return
}
@@ -228,7 +243,7 @@ func (p *AccountPages) login(w http.ResponseWriter, r *http.Request) {
ctx := account.WithClient(r.Context(), clientAddr(r))
token, csrf, err := p.accounts.SignIn(ctx, email, r.FormValue("password"))
if err != nil {
d := loginData{Email: email, SignupOpen: p.accounts.SignupOpen(), Version: p.version, CSRF: setLoginToken(w, r)}
d := loginData{SSO: p.ssoLabel(), Email: email, SignupOpen: p.accounts.SignupOpen(), Version: p.version, CSRF: setLoginToken(w, r)}
switch {
case errors.Is(err, account.ErrBadCredentials),
errors.Is(err, account.ErrNotApproved),
@@ -243,6 +258,17 @@ func (p *AccountPages) login(w http.ResponseWriter, r *http.Request) {
renderStatus(w, http.StatusUnauthorized, "login", d)
return
}
// Through the provider only, except an administrator's way in for when
// the provider itself is broken.
if p.oidc != nil && p.oidc.cfg.NoPasswords {
if _, u, err := p.accounts.Resolve(ctx, token); err != nil || !u.IsAdmin {
_ = p.accounts.SignOut(ctx, token)
renderStatus(w, http.StatusForbidden, "login", loginData{SSO: p.ssoLabel(), Email: email,
SignupOpen: p.accounts.SignupOpen(), Version: p.version, CSRF: setLoginToken(w, r),
Error: "this gateway signs people in with " + p.ssoLabel() + ", not passwords"})
return
}
}
setSessionCookie(w, r, token)
setCSRFCookie(w, r, csrf)
internal/web/admin_page.go
+1 -1
View File
@@ -20,7 +20,7 @@ func (p *AccountPages) AdminPage(dashboard http.HandlerFunc) http.HandlerFunc {
http.Redirect(w, r, "/login", http.StatusSeeOther)
return
}
if !user.IsAdmin {
if !staffMay(user, r) {
http.NotFound(w, r)
return
}
internal/web/admin_tenancy_test.go
+41
View File
@@ -90,9 +90,22 @@ func (f *fakeAdmin) AllKeys(context.Context) ([]store.KeyRow, error)
func (f *fakeAdmin) RevokeKey(context.Context, string, time.Time) error { return f.err }
func tenancyPage(t *testing.T, s *fakeAdmin, admin bool) *http.ServeMux {
t.Helper()
return tenancyPageAs(t, s, admin, nil)
}
func tenancyPageWith(t *testing.T, s *fakeAdmin, with func(*AccountPages)) *http.ServeMux {
t.Helper()
return tenancyPageAs(t, s, true, with)
}
func tenancyPageAs(t *testing.T, s *fakeAdmin, admin bool, with func(*AccountPages)) *http.ServeMux {
t.Helper()
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: admin}}
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
if with != nil {
with(p)
}
mux := http.NewServeMux()
p.RegisterAdminUsers(mux, s)
return mux
@@ -199,3 +212,31 @@ func TestThisPageSetsNoQuotas(t *testing.T) {
}
}
}
type auditMem struct{ got []store.AuditEvent }
func (a *auditMem) WriteAudit(_ context.Context, e store.AuditEvent) error {
a.got = append(a.got, e)
return nil
}
func TestAChangeOnTheAdminPagesIsAudited(t *testing.T) {
s := &fakeAdmin{}
am := &auditMem{}
mux := tenancyPageWith(t, s, func(p *AccountPages) { p.SetAudit(am) })
mirrorPost(t, mux, "/admin/tenancy/org", url.Values{"csrf": {"tok"}, "name": {"Scania"}})
s.err = store.ErrDuplicateName
mirrorPost(t, mux, "/admin/tenancy/org", url.Values{"csrf": {"tok"}, "name": {"Scania"}})
if len(am.got) != 2 {
t.Fatalf("recorded %d", len(am.got))
}
ok, refused := am.got[0], am.got[1]
if ok.Source != "web" || ok.Action != "POST /admin/tenancy/org" || ok.Actor == "" ||
!strings.Contains(ok.Detail, `"name":"Scania"`) || !strings.Contains(ok.Detail, "created") ||
strings.Contains(ok.Detail, "tok") {
t.Errorf("%+v", ok)
}
if !strings.Contains(refused.Detail, "refused") {
t.Errorf("a refusal reads as one: %+v", refused)
}
}
internal/web/admin_users.go
+6 -2
View File
@@ -420,11 +420,13 @@ func (p *AccountPages) tenancyAction(s AccountAdmin,
err = plainError(err)
p.log.Warn("a tenancy change was refused", "admin", admin.Email,
"path", r.URL.Path, "err", err)
noteAudit(r, "", "", err.Error())
p.renderUsers(w, r, s, admin, "", err.Error(), r.FormValue("open"))
return
}
p.log.Info("the tenancy tree was changed", "admin", admin.Email,
"path", r.URL.Path, "what", msg)
noteAudit(r, "", msg, "")
p.renderUsers(w, r, s, admin, msg, "", r.FormValue("open"))
})
}
@@ -440,12 +442,12 @@ func (p *AccountPages) adminOnly(
http.Redirect(w, r, "/login", http.StatusSeeOther)
return
}
if !user.IsAdmin {
if !staffMay(user, r) {
// 404 rather than 403: a 403 confirms the page exists.
http.NotFound(w, r)
return
}
next(w, p.withNav(r, user), user)
p.audited(w, p.withNav(r, user), user, func(w http.ResponseWriter, r *http.Request) { next(w, r, user) })
}
}
@@ -474,11 +476,13 @@ func (p *AccountPages) adminAction(s AccountAdmin,
// action is as worth knowing about as one that went through.
p.log.Warn("an administrative action was refused",
"admin", admin.Email, "subject", subject, "err", err)
noteAudit(r, subject, "", err.Error())
p.renderUsers(w, r, s, admin, "", subject+": "+err.Error(), r.FormValue("open"))
return
}
p.log.Info("an account was changed",
"admin", admin.Email, "subject", subject, "what", msg)
noteAudit(r, subject, msg, "")
p.renderUsers(w, r, s, admin, subject+" "+msg, "", r.FormValue("open"))
})
}
internal/web/audit.go
+85
View File
@@ -0,0 +1,85 @@
package web
import (
"context"
"net/http"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// SetAudit has every change made through these pages recorded.
func (p *AccountPages) SetAudit(s audit.Sink) { p.audit = s }
// auditNote is what a handler says about the change it made: whom it was
// about, and how it ended in words — pages answer 200 to a refusal too.
type auditNote struct{ subject, result, refused string }
type auditNoteKey struct{}
// noteAudit lets the handler of a change say whom it concerned and how it
// ended.
func noteAudit(r *http.Request, subject, result, refused string) {
if n, ok := r.Context().Value(auditNoteKey{}).(*auditNote); ok {
n.subject, n.result, n.refused = subject, result, refused
}
}
type statusRecorder struct {
http.ResponseWriter
status int
}
func (w *statusRecorder) WriteHeader(code int) {
if w.status == 0 {
w.status = code
}
w.ResponseWriter.WriteHeader(code)
}
func (w *statusRecorder) Write(b []byte) (int, error) {
if w.status == 0 {
w.status = http.StatusOK
}
return w.ResponseWriter.Write(b)
}
// audited runs a change and records it: who, which page, the form with its
// secrets taken out, and how it ended.
func (p *AccountPages) audited(w http.ResponseWriter, r *http.Request, actor store.User,
run func(w http.ResponseWriter, r *http.Request),
) {
if p.audit == nil || r.Method == http.MethodGet || r.Method == http.MethodHead {
run(w, r)
return
}
note := &auditNote{}
r = r.WithContext(context.WithValue(r.Context(), auditNoteKey{}, note))
sw := &statusRecorder{ResponseWriter: w}
run(sw, r)
fields := audit.FormFields(r.PostForm)
if note.result != "" {
fields["result"] = note.result
}
if note.refused != "" {
fields["refused"] = note.refused
}
audit.Write(r.Context(), p.audit, p.log, store.AuditEvent{
At: time.Now(), Actor: actor.Email, Source: "web", Action: r.Method + " " + r.URL.Path,
Target: note.subject, Detail: audit.Redact(fields), Status: sw.status,
})
}
// selfAudited records a change a person makes to their own account: a key
// made or revoked, a password changed.
func (p *AccountPages) selfAudited(h http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
_, user, ok := p.session(r)
if !ok {
h(w, r)
return
}
p.audited(w, r, user, h)
}
}
internal/web/nav.go
+5 -2
View File
@@ -17,8 +17,10 @@ type Nav struct {
// Current is the page being shown, so its link is marked rather than
// offered: "cluster", "people", "quotas", "usage", "mirror", "chat", "keys".
Current string
// Admin shows the administration links.
// Admin shows the administration links, and Role which of them: a
// billing role sees usage and quotas only.
Admin bool
Role string
// Mirror and Quotas are whether those pages were mounted at all; usage is
// mounted with quotas.
Mirror, Quotas bool
@@ -37,7 +39,8 @@ type navKey struct{}
func (p *AccountPages) navFor(r *http.Request, user store.User, current string) Nav {
n := Nav{
Current: current,
Admin: user.IsAdmin,
Admin: user.Staff(),
Role: user.StaffRole(),
Mirror: p.hasMirror,
Quotas: p.hasQuotas,
Guard: p.hasGuard,
internal/web/oidc.go
+375
View File
@@ -0,0 +1,375 @@
package web
import (
"context"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"net/http"
"os"
"slices"
"strings"
"sync"
"time"
"github.com/coreos/go-oidc/v3/oidc"
"golang.org/x/oauth2"
"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// OIDCStore is what signing in through a provider needs of the store.
type OIDCStore interface {
UserByEmail(ctx context.Context, email string) (store.User, error)
CreateUser(ctx context.Context, u store.User) error
SetStaffRole(ctx context.Context, id, role string) error
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
AllOrgs(ctx context.Context) ([]store.Org, error)
AllTeams(ctx context.Context) ([]store.TeamPath, error)
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
}
// oidcLogin is sign-in through an identity provider.
type oidcLogin struct {
cfg config.OIDC
secret string
st OIDCStore
key []byte // signs the cookie that carries a sign-in across the provider
mu sync.Mutex
provider *oidc.Provider
}
const oidcCookie = "llmgw_oidc"
// RegisterOIDC mounts /login/oidc and its callback, and puts the button on
// the sign-in page.
func (p *AccountPages) RegisterOIDC(mux *http.ServeMux, cfg config.OIDC, st OIDCStore) error {
secret := ""
if cfg.ClientSecretFile != "" {
b, err := os.ReadFile(cfg.ClientSecretFile)
if err != nil {
return fmt.Errorf("auth.oidc.client_secret_file: %w", err)
}
secret = strings.TrimSpace(string(b))
}
key := make([]byte, 32)
if _, err := rand.Read(key); err != nil {
return err
}
o := &oidcLogin{cfg: cfg, secret: secret, st: st, key: key}
p.oidc = o
mux.HandleFunc("GET /login/oidc", p.oidcStart)
mux.HandleFunc("GET /login/oidc/callback", p.oidcCallback)
return nil
}
// providerFor discovers the provider the first time it is needed, and again
// after a failure: a provider down when the gateway started is no reason for
// sign-in never to work.
func (o *oidcLogin) providerFor(ctx context.Context) (*oidc.Provider, error) {
o.mu.Lock()
defer o.mu.Unlock()
if o.provider != nil {
return o.provider, nil
}
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
pr, err := oidc.NewProvider(ctx, o.cfg.Issuer)
if err != nil {
return nil, err
}
o.provider = pr
return pr, nil
}
func (o *oidcLogin) oauth(pr *oidc.Provider, r *http.Request) *oauth2.Config {
redirect := o.cfg.RedirectURL
if redirect == "" {
redirect = baseURL(r) + "/login/oidc/callback"
}
scopes := []string{oidc.ScopeOpenID}
if len(o.cfg.Scopes) > 0 {
scopes = append(scopes, o.cfg.Scopes...)
} else {
scopes = append(scopes, "email", "profile")
if len(o.cfg.Roles) > 0 || len(o.cfg.Memberships) > 0 {
scopes = append(scopes, "groups")
}
}
return &oauth2.Config{ClientID: o.cfg.ClientID, ClientSecret: o.secret, Endpoint: pr.Endpoint(),
RedirectURL: redirect, Scopes: scopes}
}
// The cookie: state, nonce and PKCE verifier, and when, signed.
func (o *oidcLogin) seal(parts ...string) string {
v := strings.Join(parts, ".")
m := hmac.New(sha256.New, o.key)
m.Write([]byte(v))
return v + "." + hex.EncodeToString(m.Sum(nil))
}
func (o *oidcLogin) open(c string) ([]string, bool) {
i := strings.LastIndex(c, ".")
if i < 0 {
return nil, false
}
v, sig := c[:i], c[i+1:]
m := hmac.New(sha256.New, o.key)
m.Write([]byte(v))
want := hex.EncodeToString(m.Sum(nil))
if !hmac.Equal([]byte(sig), []byte(want)) {
return nil, false
}
return strings.Split(v, "."), true
}
func randomToken() string {
b := make([]byte, 24)
_, _ = rand.Read(b)
return base64.RawURLEncoding.EncodeToString(b)
}
func (p *AccountPages) oidcStart(w http.ResponseWriter, r *http.Request) {
noStore(w)
o := p.oidc
pr, err := o.providerFor(r.Context())
if err != nil {
p.log.Warn("the identity provider did not answer", "issuer", o.cfg.Issuer, "err", err)
renderStatus(w, http.StatusBadGateway, "login", loginData{SignupOpen: p.accounts.SignupOpen(), Version: p.version,
CSRF: setLoginToken(w, r), SSO: o.cfg.Label, Error: "the sign-in service did not answer; try again shortly"})
return
}
state, nonce, verifier := randomToken(), randomToken(), oauth2.GenerateVerifier()
http.SetCookie(w, &http.Cookie{Name: oidcCookie, Value: o.seal(state, nonce, verifier,
fmt.Sprint(time.Now().Unix())), Path: "/login/oidc", MaxAge: 600, HttpOnly: true,
Secure: isHTTPS(r), SameSite: http.SameSiteLaxMode})
http.Redirect(w, r, o.oauth(pr, r).AuthCodeURL(state, oidc.Nonce(nonce), oauth2.S256ChallengeOption(verifier)),
http.StatusFound)
}
// errOIDC is a sign-in through the provider that did not work, said plainly.
var errOIDC = errors.New("signing in through the provider did not work; start again")
func (p *AccountPages) oidcCallback(w http.ResponseWriter, r *http.Request) {
noStore(w)
o := p.oidc
fail := func(why string, err error) {
p.log.Warn("an OIDC sign-in was refused", "why", why, "err", err)
http.SetCookie(w, &http.Cookie{Name: oidcCookie, Path: "/login/oidc", MaxAge: -1})
msg := errOIDC.Error()
if errors.Is(err, errOIDCAccount) {
msg = err.Error()
}
renderStatus(w, http.StatusForbidden, "login", loginData{SignupOpen: p.accounts.SignupOpen(),
Version: p.version, CSRF: setLoginToken(w, r), SSO: o.cfg.Label, Error: msg})
}
c, err := r.Cookie(oidcCookie)
if err != nil {
fail("no sign-in under way", err)
return
}
parts, ok := o.open(c.Value)
if !ok || len(parts) != 4 || r.URL.Query().Get("state") != parts[0] {
fail("state", nil)
return
}
if e := r.URL.Query().Get("error"); e != "" {
fail("the provider said "+e, nil)
return
}
pr, err := o.providerFor(r.Context())
if err != nil {
fail("provider", err)
return
}
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
tok, err := o.oauth(pr, r).Exchange(ctx, r.URL.Query().Get("code"), oauth2.VerifierOption(parts[2]))
if err != nil {
fail("code exchange", err)
return
}
raw, _ := tok.Extra("id_token").(string)
idt, err := pr.Verifier(&oidc.Config{ClientID: o.cfg.ClientID}).Verify(ctx, raw)
if err != nil {
fail("id token", err)
return
}
if idt.Nonce != parts[1] {
fail("nonce", nil)
return
}
var claims map[string]any
if err := idt.Claims(&claims); err != nil {
fail("claims", err)
return
}
user, err := p.oidcUser(ctx, claims)
if err != nil {
fail("account", err)
return
}
token, csrf, err := p.accounts.OpenSession(ctx, user.ID)
if err != nil {
fail("session", err)
return
}
http.SetCookie(w, &http.Cookie{Name: oidcCookie, Path: "/login/oidc", MaxAge: -1})
setSessionCookie(w, r, token)
setCSRFCookie(w, r, csrf)
audit.Write(r.Context(), p.audit, p.log, store.AuditEvent{At: time.Now(), Actor: user.Email, Source: "web",
Action: "sign in through " + o.cfg.Issuer, Status: http.StatusSeeOther,
Detail: audit.Redact(map[string]any{"role": user.StaffRole()})})
http.Redirect(w, r, "/profile", http.StatusSeeOther)
}
// errOIDCAccount is a provider's person the gateway will not let in, with why.
var errOIDCAccount = errors.New("")
type accountError string
func (e accountError) Error() string { return string(e) }
func (e accountError) Is(t error) bool { return t == errOIDCAccount }
// oidcUser is the account a provider's person signs in as: found by their
// verified email, made when the gateway makes accounts this way, with the
// staff role and the memberships their groups give.
func (p *AccountPages) oidcUser(ctx context.Context, claims map[string]any) (store.User, error) {
o := p.oidc
email, _ := claims["email"].(string)
email = store.NormalizeEmail(email)
if email == "" {
return store.User{}, accountError("the provider gave no email address; ask it to send the email claim")
}
if v, ok := claims["email_verified"].(bool); ok && !v {
return store.User{}, accountError("the provider has not verified " + email)
}
u, err := o.st.UserByEmail(ctx, email)
switch {
case errors.Is(err, store.ErrNotFound) && o.cfg.CreateAccounts:
name, _ := claims["name"].(string)
if name == "" {
name = email
}
u = store.User{ID: "usr-" + strings.ToLower(randomToken()[:16]), Email: email, Name: name,
// No password works: this account signs in through the provider.
PasswordHash: "!oidc", State: store.UserActive, CreatedAt: time.Now()}
if err := o.st.CreateUser(ctx, u); err != nil {
return store.User{}, err
}
p.log.Info("an account was made for a person the identity provider vouched for", "email", email)
case errors.Is(err, store.ErrNotFound):
return store.User{}, accountError("there is no account for " + email + " on this gateway; ask an administrator for one")
case err != nil:
return store.User{}, err
}
if !u.CanSignIn() {
return store.User{}, accountError("the account for " + email + " is " + string(u.State))
}
groups := claimStrings(claims[or(o.cfg.GroupsClaim, "groups")])
if len(o.cfg.Roles) > 0 {
role := ""
for _, r := range []string{store.RoleAdmin, store.RoleOperator, store.RoleBilling, store.RoleViewer} {
if slices.ContainsFunc(o.cfg.Roles[r], func(g string) bool { return slices.Contains(groups, g) }) {
role = r
break
}
}
if role != u.StaffRole() {
if err := o.st.SetStaffRole(ctx, u.ID, role); err != nil {
return store.User{}, err
}
u.IsAdmin, u.Role = role == store.RoleAdmin, role
if role == store.RoleAdmin {
u.Role = ""
}
}
}
for _, g := range groups {
where, ok := o.cfg.Memberships[g]
if !ok {
continue
}
scope, id, err := p.oidcPlace(ctx, where)
if err != nil {
p.log.Warn("a group maps to a place the tree does not have", "group", g, "place", where, "err", err)
continue
}
if err := o.st.AddMembership(ctx, u.ID, scope, id, store.RoleMember, "oidc", time.Now()); err != nil {
p.log.Warn("could not add a membership from a group", "group", g, "err", err)
}
}
return u, nil
}
// oidcPlace finds org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT.
func (p *AccountPages) oidcPlace(ctx context.Context, where string) (store.Scope, string, error) {
kind, path, _ := strings.Cut(where, ":")
// The store writes paths "Acme / ml / train"; a mapping writes them
// either way.
norm := func(s string) string {
parts := strings.Split(s, "/")
for i := range parts {
parts[i] = strings.TrimSpace(parts[i])
}
return strings.Join(parts, "/")
}
same := func(a string) bool { return strings.EqualFold(norm(a), norm(path)) }
switch kind {
case "org":
orgs, err := p.oidc.st.AllOrgs(ctx)
for _, o := range orgs {
if same(o.Name) {
return store.ScopeOrg, o.ID, nil
}
}
return "", "", errors.Join(err, errors.New("no such organization"))
case "team":
teams, err := p.oidc.st.AllTeams(ctx)
for _, t := range teams {
if same(t.OrgName + "/" + t.Name) {
return store.ScopeTeam, t.ID, nil
}
}
return "", "", errors.Join(err, errors.New("no such team"))
default:
projects, err := p.oidc.st.AllProjects(ctx)
for _, pp := range projects {
if same(pp.Path) {
return store.ScopeProject, pp.ID, nil
}
}
return "", "", errors.Join(err, errors.New("no such project"))
}
}
func claimStrings(v any) []string {
switch t := v.(type) {
case []any:
var out []string
for _, x := range t {
if s, ok := x.(string); ok {
out = append(out, s)
}
}
return out
case string:
return []string{t}
}
return nil
}
func or(s, def string) string {
if s == "" {
return def
}
return s
}
internal/web/oidc_test.go
+174
View File
@@ -0,0 +1,174 @@
package web
import (
"context"
"crypto/rand"
"crypto/rsa"
"encoding/json"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
jose "github.com/go-jose/go-jose/v4"
"github.com/go-jose/go-jose/v4/jwt"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// fakeIdP is an OpenID provider that issues an ID token with whatever
// claims the test sets.
type fakeIdP struct {
srv *httptest.Server
key *rsa.PrivateKey
claims map[string]any
nonce string
}
func newFakeIdP(t *testing.T) *fakeIdP {
t.Helper()
k, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
t.Fatal(err)
}
f := &fakeIdP{key: k}
mux := http.NewServeMux()
f.srv = httptest.NewServer(mux)
t.Cleanup(f.srv.Close)
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"issuer": f.srv.URL,
"authorization_endpoint": f.srv.URL + "/authorize", "token_endpoint": f.srv.URL + "/token",
"jwks_uri": f.srv.URL + "/jwks", "id_token_signing_alg_values_supported": []string{"RS256"}})
})
mux.HandleFunc("/jwks", func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(jose.JSONWebKeySet{Keys: []jose.JSONWebKey{
{Key: &k.PublicKey, KeyID: "k1", Algorithm: "RS256", Use: "sig"}}})
})
mux.HandleFunc("/token", func(w http.ResponseWriter, r *http.Request) {
_ = r.ParseForm()
if r.Form.Get("code_verifier") == "" {
http.Error(w, "pkce", 400)
return
}
signer, _ := jose.NewSigner(jose.SigningKey{Algorithm: jose.RS256, Key: k},
(&jose.SignerOptions{}).WithHeader("kid", "k1"))
c := map[string]any{"iss": f.srv.URL, "aud": "llmgw", "sub": "s1", "nonce": f.nonce,
"iat": time.Now().Unix(), "exp": time.Now().Add(time.Minute).Unix()}
for k, v := range f.claims {
c[k] = v
}
tok, _ := jwt.Signed(signer).Claims(c).Serialize()
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"access_token": "a", "token_type": "Bearer", "id_token": tok})
})
return f
}
type oidcStoreMem struct {
users map[string]store.User
roles map[string]string
joined []string
}
func (m *oidcStoreMem) UserByEmail(_ context.Context, e string) (store.User, error) {
if u, ok := m.users[e]; ok {
return u, nil
}
return store.User{}, store.ErrNotFound
}
func (m *oidcStoreMem) CreateUser(_ context.Context, u store.User) error {
m.users[u.Email] = u
return nil
}
func (m *oidcStoreMem) SetStaffRole(_ context.Context, id, role string) error {
m.roles[id] = role
return nil
}
func (m *oidcStoreMem) AddMembership(_ context.Context, user string, scope store.Scope, id, _, _ string, _ time.Time) error {
m.joined = append(m.joined, user+"@"+string(scope)+":"+id)
return nil
}
func (m *oidcStoreMem) AllOrgs(context.Context) ([]store.Org, error) { return nil, nil }
func (m *oidcStoreMem) AllTeams(context.Context) ([]store.TeamPath, error) {
return []store.TeamPath{{ID: "t1", Name: "ml", OrgName: "Acme"}}, nil
}
func (m *oidcStoreMem) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
type sessionAccounts struct{ fakeAccounts }
func (sessionAccounts) OpenSession(context.Context, string) (string, string, error) {
return "sess", "csrf", nil
}
// signIn goes through /login/oidc and back, as a browser would.
func signIn(t *testing.T, mux http.Handler, idp *fakeIdP, tamper bool) *httptest.ResponseRecorder {
t.Helper()
start := httptest.NewRecorder()
mux.ServeHTTP(start, httptest.NewRequest("GET", "/login/oidc", nil))
if start.Code != http.StatusFound {
t.Fatalf("start: %d %s", start.Code, start.Body.String())
}
to, _ := url.Parse(start.Header().Get("Location"))
if !strings.HasPrefix(to.String(), idp.srv.URL+"/authorize") || to.Query().Get("code_challenge") == "" {
t.Fatalf("redirected to %s", to)
}
idp.nonce = to.Query().Get("nonce")
state := to.Query().Get("state")
if tamper {
state = "someone-elses"
}
cb := httptest.NewRequest("GET", "/login/oidc/callback?code=c1&state="+state, nil)
for _, c := range start.Result().Cookies() {
cb.AddCookie(c)
}
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, cb)
return rec
}
func TestSigningInThroughAProvider(t *testing.T) {
idp := newFakeIdP(t)
st := &oidcStoreMem{users: map[string]store.User{}, roles: map[string]string{}}
p := NewAccountPages(sessionAccounts{}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
mux := http.NewServeMux()
if err := p.RegisterOIDC(mux, config.OIDC{Issuer: idp.srv.URL, ClientID: "llmgw", CreateAccounts: true,
Roles: map[string][]string{"operator": {"ops"}, "admin": {"root"}},
Memberships: map[string]string{"ml": "team:Acme/ml"}}, st); err != nil {
t.Fatal(err)
}
idp.claims = map[string]any{"email": "Ann@Example.com", "email_verified": true, "name": "Ann", "groups": []string{"ops", "ml"}}
rec := signIn(t, mux, idp, false)
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/profile" {
t.Fatalf("callback: %d %s", rec.Code, rec.Body.String())
}
u, ok := st.users["ann@example.com"]
if !ok || u.State != store.UserActive || st.roles[u.ID] != "operator" || len(st.joined) != 1 || !strings.HasSuffix(st.joined[0], "team:t1") {
t.Errorf("account %+v roles %v joined %v", u, st.roles, st.joined)
}
if !strings.Contains(rec.Header().Get("Set-Cookie")+strings.Join(rec.Header().Values("Set-Cookie"), ";"), "sess") {
t.Error("no session cookie")
}
// Refused: a state that is not this browser's, an unverified address,
// and nobody the gateway knows when it does not make accounts.
if rec := signIn(t, mux, idp, true); rec.Code != http.StatusForbidden {
t.Errorf("a tampered state: %d", rec.Code)
}
idp.claims = map[string]any{"email": "bob@example.com", "email_verified": false}
if rec := signIn(t, mux, idp, false); rec.Code != http.StatusForbidden || !strings.Contains(rec.Body.String(), "not verified") {
t.Errorf("unverified: %d", rec.Code)
}
p2 := NewAccountPages(sessionAccounts{}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
mux2 := http.NewServeMux()
_ = p2.RegisterOIDC(mux2, config.OIDC{Issuer: idp.srv.URL, ClientID: "llmgw"}, st)
idp.claims = map[string]any{"email": "carol@example.com"}
if rec := signIn(t, mux2, idp, false); rec.Code != http.StatusForbidden || !strings.Contains(rec.Body.String(), "no account for carol@example.com") {
t.Errorf("unknown without create_accounts: %d %s", rec.Code, rec.Body.String())
}
}
internal/web/roles.go
+33
View File
@@ -0,0 +1,33 @@
package web
import (
"net/http"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// staffMay reports whether a person may do this to an administration page.
//
// An administrator may do anything. A viewer may look at every page and
// change nothing. An operator may also run the cluster from them: models
// through the mirror, and a node's engines. Billing may look at usage and
// quotas. Everything that touches people, the tree, limits or keys stays
// with administrators.
func staffMay(u store.User, r *http.Request) bool {
if u.IsAdmin {
return true
}
read := r.Method == http.MethodGet || r.Method == http.MethodHead
path := r.URL.Path
switch u.Role {
case store.RoleViewer:
return read
case store.RoleOperator:
return read || strings.HasPrefix(path, "/admin/mirror/") ||
strings.HasPrefix(path, "/admin/nodes/") && strings.HasSuffix(path, "/engines")
case store.RoleBilling:
return read && (path == "/admin/usage" || path == "/admin/quotas")
}
return false
}
internal/web/roles_test.go
+48
View File
@@ -0,0 +1,48 @@
package web
import (
"net/http/httptest"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func TestEachStaffRoleMayDoWhatItSays(t *testing.T) {
cases := []struct {
role, method, path string
may bool
}{
{store.RoleViewer, "GET", "/admin/users", true},
{store.RoleViewer, "POST", "/admin/users/u1/suspend", false},
{store.RoleViewer, "POST", "/admin/mirror/fetch", false},
{store.RoleOperator, "GET", "/admin", true},
{store.RoleOperator, "POST", "/admin/mirror/fetch", true},
{store.RoleOperator, "POST", "/admin/nodes/n1/engines", true},
{store.RoleOperator, "POST", "/admin/quotas", false},
{store.RoleOperator, "POST", "/admin/tenancy/org", false},
{store.RoleBilling, "GET", "/admin/usage", true},
{store.RoleBilling, "GET", "/admin/quotas", true},
{store.RoleBilling, "GET", "/admin/users", false},
{store.RoleBilling, "POST", "/admin/quotas", false},
{"", "GET", "/admin", false},
}
for _, c := range cases {
u := store.User{Role: c.role}
if got := staffMay(u, httptest.NewRequest(c.method, c.path, nil)); got != c.may {
t.Errorf("%s %s %s: %v", c.role, c.method, c.path, got)
}
}
if !staffMay(store.User{IsAdmin: true}, httptest.NewRequest("POST", "/admin/users/u1/delete", nil)) {
t.Error("an administrator may do anything")
}
}
func TestABillingRoleIsShownUsageAndQuotasOnly(t *testing.T) {
rec := httptest.NewRecorder()
render(rec, "admin_guard", guardPageData{Nav: Nav{Admin: true, Role: store.RoleBilling}})
body := rec.Body.String()
if !strings.Contains(body, `href="/admin/usage"`) || strings.Contains(body, `href="/admin/users"`) {
t.Errorf("menu: %s", body)
}
}
internal/web/templates/account.html
+1
View File
@@ -46,6 +46,7 @@ an operator to reset it.</p>
<label>Password<input type="password" name="password" required autocomplete="current-password"></label>
<button type="submit">Sign in</button>
</form>
{{if .SSO}}<p><a class="btn" href="/login/oidc">Sign in with {{.SSO}}</a></p>{{end}}
{{if .SignupOpen}}<p class="note">No account? <a href="/signup">Sign up</a>.</p>{{end}}
<footer><span>llmgw {{.Version}}</span><span><a href="/">about this gateway</a></span></footer>
</div></body></html>{{end}}
internal/web/templates/nav.html
+5 -1
View File
@@ -6,7 +6,11 @@
<a href="/org"{{if eq .Current "org"}} aria-current="page"{{end}}>Organization</a>
<span class="menu-gap" aria-hidden="true"></span>
{{- end}}
{{- if .Admin}}
{{- if and .Admin (eq .Role "billing")}}
<a href="/admin/usage"{{if eq .Current "usage"}} aria-current="page"{{end}}>Usage</a>
<a href="/admin/quotas"{{if eq .Current "quotas"}} aria-current="page"{{end}}>Quotas</a>
<span class="menu-gap" aria-hidden="true"></span>
{{- else if .Admin}}
<a href="/admin"{{if eq .Current "cluster"}} aria-current="page"{{end}}>Cluster</a>
{{- /* Limits and usage are tabs of the people page, not pages of their own. */}}
<a href="/admin/users"{{if eq .Current "people"}} aria-current="page"{{end}}>People</a>