llmgw/docs/admin-api.md

# The admin API

Written for whoever automates a cluster rather than typing at it: how to reach
the control plane's management API, what it asks of a request, and where its
description is.

`llmgw ctl` and the console are clients of this API and nothing more, so
anything they do can be done with `curl`, a generated client or the
[Terraform provider](../terraform-provider-llmgw/README.md). It adds and
removes models, scales them, cordons and upgrades nodes, sets capacity
quotas, attaches guardrail policies, adds organizations, teams and projects,
issues and revokes API keys, puts people in the tree, and reads the fleet,
usage and the audit log.

## Reaching it

The API is under `/api/v1` on the admin listener, `admin_listen` in
`/etc/llmgw/server.yaml`: `127.0.0.1:9090` unless changed. It is on loopback
on purpose. The same listener serves pprof when it is turned on, and the
server refuses to start with it on any other address unless
`admin_remote: true` — the token would cross the network in clear.

From another machine, tunnel to it rather than moving it:

```sh
ssh -N -L 9090:127.0.0.1:9090 admin@control-plane &
curl -s -H "Authorization: Bearer $LLMGW_ADMIN_TOKEN" http://127.0.0.1:9090/api/v1/summary
```

With `admin_remote: true` and `admin_listen: 10.0.0.5:9090`, call that address
directly — on a network you trust, or behind a TLS proxy that keeps the
`Host` header an address.

## The token

Every path under `/api` needs the admin token as a bearer token:

```
Authorization: Bearer 6f1c…
```

The server writes it the first time it starts, 0600, to
`/var/lib/llmgw/admin-token` under its systemd unit (`admin_token_file`
moves it; a server run by hand keeps it in `secrets/admin-token` beside its
configuration). `llmgw ctl` reads it from there, which is why it runs with
sudo, or from `LLMGW_ADMIN_TOKEN`. To give it to automation:

```sh
sudo cat /var/lib/llmgw/admin-token
```

The token is the whole of the API's authorisation: whoever holds it can do
everything the API does. Keep it where you keep other root credentials.

## What a request must look like

Besides the token, the listener refuses three things before a handler sees
them, as defences against a browser being turned against it:

- **A name in `Host`.** The listener answers to an address or `localhost`,
  not a DNS name, so a page on some other site cannot reach it by rebinding
  a name to 127.0.0.1. Call it as `http://127.0.0.1:9090`, not
  `http://cp.example.com:9090`; behind a proxy, have the proxy send an
  address.
- **A cross-site request.** One whose `Origin` is another host, or whose
  `Sec-Fetch-Site` says another site sent it.
- **A change that is not JSON.** Every method but `GET` and `HEAD` must carry
  `Content-Type: application/json`, even a `DELETE` or a `POST` with no body.
  A form post from a page cannot set that header.

A change is recorded in the audit log with who made it, as the request says
in `X-Llmgw-Actor` (letters, digits and `._-@`, at most 64; `unknown` when
absent). `llmgw ctl` sends the name of whoever ran it; automation should send
its own, `terraform` or `ci@deploy`, so `llmgw ctl audit` can tell them apart.

A refusal is a 4xx or 5xx with `{"error": "…"}` saying why. 501 means this
control plane lacks what the endpoint needs — a store, a mirror, guard
policies — and says which.

## Some calls

```sh
H=(-H "Authorization: Bearer $LLMGW_ADMIN_TOKEN" -H "Content-Type: application/json" -H "X-Llmgw-Actor: $USER")
API=http://127.0.0.1:9090/api/v1

# The fleet in one answer, and every model.
curl -s "${H[@]}" $API/summary
curl -s "${H[@]}" $API/models

# Add a model, read back its definition with the defaults filled in, scale it.
curl -s "${H[@]}" -X POST $API/models -d '{
  "name": "qwen-small",
  "source": "/srv/models/qwen2.5-1.5b-instruct-q4_k_m.gguf",
  "profiles": [{"name": "cpu", "engine": "llamacpp", "min_memory_gb": 4, "cpu_cores": 4}],
  "scaling": {"min": 1, "max": 2}
}'
curl -s "${H[@]}" $API/models/qwen-small
curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2, "idle_timeout": 1800000000000}'

# A team's capacity quota.
curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'

# An organization, a team in it and a project in that; then a key for the
# project. The answer holds the key once: only its hash is kept.
curl -s "${H[@]}" -X POST $API/orgs -d '{"name": "Acme"}'
curl -s "${H[@]}" -X POST $API/teams -d '{"name": "ml", "in": "Acme"}'
curl -s "${H[@]}" -X POST $API/projects -d '{"name": "search", "in": "Acme/ml"}'
curl -s "${H[@]}" -X POST $API/keys -d '{"name": "ci", "project": "Acme/ml/search", "expires_in": "720h"}'
curl -s "${H[@]}" -X POST $API/users/ayse@example.com/memberships -d '{"target": "team:Acme/ml"}'

# Remove the model; a DELETE carries the content type too.
curl -s "${H[@]}" -X DELETE $API/models/qwen-small
```

Durations in bodies are integer nanoseconds, as Go encodes them: half an hour
is `1800000000000`.

## The description

The API is described in OpenAPI 3.1:

- served by every control plane at `GET /api/v1/openapi.json` (with the
  token, like the rest), as built into that binary, so it describes the
  server you are talking to;
- committed as [admin-api.openapi.json](admin-api.openapi.json) for reading
  without a cluster.

```sh
curl -s "${H[@]}" $API/openapi.json > llmgw-admin.json
```

It lists every endpoint, its path and query parameters, and the schema of
each request and answer, with the meaning of each field. Load it into any
OpenAPI tool — Swagger UI, Redoc, an editor — or generate a client from it:

```sh
docker run --rm -v "$PWD:/w" openapitools/openapi-generator-cli generate \
  -i /w/llmgw-admin.json -g python -o /w/llmgw-admin-client
```

### How it is kept true

The document is generated, not written. `internal/control/adminapi/routes.go`
holds a table of every endpoint: its method and path, a summary, its query
parameters, and Go values of the types it reads and answers with. The
generator in `internal/docgen/openapi` reads those types by reflection, the
way `encoding/json` encodes them, and takes each field's description from its
comment in the source.

Two tests keep it from drifting. One registers the API on a recording mux and
fails when an endpoint is served and not in the table, or in the table and not
served. The other regenerates the document and fails when it differs from the
committed one. After changing an endpoint or a type it carries:

```sh
go generate ./internal/docgen
```

## What it does not cover

- **Making people's accounts.** People sign up, are invited, or come from
  the identity provider (OIDC, SCIM); the API lists them and puts them in
  the tree, and `llmgw users` approves, disables and deletes them. Key
  lifetime limits and spending limits are `llmgw keys ttl` and `llmgw keys
  limits`.
- **The client-facing API** (`/v1/chat/completions` and the rest), which is
  OpenAI's and Anthropic's, on `listen`, with API keys.
- **The rest of the admin listener**: `/metrics`, `/health`, the dashboard
  pages and `/debug/pprof`, which are not under `/api`.