llmgw_org, llmgw_team and llmgw_project rename and move in place; llmgw_key holds the key in a sensitive secret and is made again when revoked or expired outside Terraform; llmgw_membership puts a person in the tree. Tried with OpenTofu against the fake admin API, drift and import included. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
145 lines
7.2 KiB
Markdown
145 lines
7.2 KiB
Markdown
# terraform-provider-llmgw
|
|
|
|
A Terraform (and OpenTofu) provider for llmgw: the models in a cluster's
|
|
catalog, capacity quotas on its teams and projects, and where its guardrail
|
|
policies apply, kept in configuration and reviewed like the rest of it.
|
|
|
|
It talks to the control plane's admin API with the admin token, the same API
|
|
`llmgw ctl` uses; see [docs/admin-api.md](../docs/admin-api.md) for how that
|
|
API is reached and what it asks of a request.
|
|
|
|
## What it manages
|
|
|
|
| | |
|
|
|---|---|
|
|
| `llmgw_model` | A model added to the catalog at runtime, as `llmgw ctl model add` adds one. Scaling, class, owner and adapters change in place; any other change to the definition replaces the model, which stops its replicas while it is added again. |
|
|
| `llmgw_quota` | A capacity quota on an organization, team or project: a guarantee, a limit, or both, in GPUs, CPU cores and memory. |
|
|
| `llmgw_guard_attachment` | A guardrail policy attached to an organization, team, project, key or person. The policies are defined in `server.yaml`'s `guard:` section; this says where they apply. |
|
|
| `llmgw_org`, `llmgw_team`, `llmgw_project` | The tenancy tree. A team names its organization and a project its team, by path or ID; a new name renames in place, a new parent moves it with everything under it. Only something empty is removed, and a project that has ever had a key is never empty (the revoked key stays for usage history): rename it rather than destroy it. |
|
|
| `llmgw_key` | An API key in a project, for a person in a project they belong to, or a person's own. The key is in `secret` (sensitive) from the apply that made it; the gateway keeps only a hash. Any change makes a new key, and one revoked or expired outside Terraform is made again. Destroying it revokes it. |
|
|
| `llmgw_membership` | A person in an organization, team or project, as member or manager. People are not made here: they sign up, are invited, or come from the identity provider. Taking a membership away revokes their keys in projects they no longer reach. |
|
|
| `data.llmgw_models` | Every model in the catalog, with its replicas now. |
|
|
| `data.llmgw_nodes` | Every node, with its cards and state now. |
|
|
|
|
Not here: people's accounts (sign-up, invitation, OIDC/SCIM, and `llmgw
|
|
users` to approve, disable or delete), and key lifetime and spending limits
|
|
(`llmgw keys ttl`, `llmgw keys limits`). A quota or an attachment names a
|
|
part of the tree by its path or its ID; refer to an `llmgw_team`'s `id` to
|
|
have Terraform make the team first.
|
|
|
|
Models declared in the control plane's `server.yaml` are not managed here
|
|
either: the cluster refuses to remove or change them through the API, since
|
|
the configuration file would put them back on the next restart.
|
|
|
|
## Configuring it
|
|
|
|
```hcl
|
|
terraform {
|
|
required_providers {
|
|
llmgw = { source = "gitea.mynodes.xyz/ilker/llmgw" }
|
|
}
|
|
}
|
|
|
|
provider "llmgw" {
|
|
endpoint = "http://127.0.0.1:9090" # or LLMGW_ADMIN_URL
|
|
# token = "..." # or LLMGW_ADMIN_TOKEN
|
|
token_file = "/var/lib/llmgw/admin-token" # read when no token is given
|
|
actor = "terraform" # who the audit log says made the change
|
|
}
|
|
```
|
|
|
|
All four are optional and default to what is shown. The admin listener is on
|
|
loopback, so run Terraform on the control plane — as a user that can read the
|
|
token file, which is root's — or from elsewhere through a tunnel:
|
|
|
|
```sh
|
|
ssh -N -L 9090:127.0.0.1:9090 admin@control-plane &
|
|
export LLMGW_ADMIN_TOKEN=$(ssh admin@control-plane sudo cat /var/lib/llmgw/admin-token)
|
|
terraform apply
|
|
```
|
|
|
|
The endpoint must be an address, not a name: the admin listener refuses a
|
|
`Host` that is a DNS name.
|
|
|
|
[examples/main.tf](examples/main.tf) has a model on CPUs, one on GPUs, a team's
|
|
quota, a policy attachment and the two data sources.
|
|
|
|
## Installing it
|
|
|
|
It is not in a public registry. Build it and point Terraform at it, either as
|
|
a development override:
|
|
|
|
```sh
|
|
cd terraform-provider-llmgw
|
|
go build -o ~/.terraform.d/llmgw/ .
|
|
cat >> ~/.terraformrc <<'EOF'
|
|
provider_installation {
|
|
dev_overrides {
|
|
"gitea.mynodes.xyz/ilker/llmgw" = "/home/you/.terraform.d/llmgw"
|
|
}
|
|
direct {}
|
|
}
|
|
EOF
|
|
```
|
|
|
|
or in a filesystem mirror, which `terraform init` installs from like a
|
|
registry and records in the lock file:
|
|
|
|
```sh
|
|
v=0.1.0
|
|
d=~/.terraform.d/plugins/gitea.mynodes.xyz/ilker/llmgw/$v/$(go env GOOS)_$(go env GOARCH)
|
|
mkdir -p $d
|
|
go build -ldflags "-X main.version=$v" -o $d/terraform-provider-llmgw_v$v .
|
|
```
|
|
|
|
with `version = "0.1.0"` beside `source` in `required_providers`.
|
|
|
|
## How the state behaves
|
|
|
|
- **A model's definition is kept as written.** The catalog fills in what a
|
|
definition leaves out — `tp = 1`, `engine = "vllm"`, a 20-minute start-up
|
|
timeout — and the state does not take those values, so a plan does not show
|
|
them as differences. What can change in place is read back from the
|
|
cluster: a model scaled or reclassified from a terminal shows as a change to
|
|
put right on the next plan, and one removed from a terminal is added again.
|
|
- **`scaling.max` left out follows `min`**, as in the configuration file, and
|
|
is read back. Leaving the whole `scaling` block out means no replicas: the
|
|
model is in the catalog and does not run.
|
|
- **Durations are written as Go writes them**, `"30m"` or `"1h30m"`.
|
|
- **An owner or a quota's scope is kept as written** (`team:Acme/ml`) beside
|
|
the ID the cluster resolved it to (`owner_id`, and the quota's `id`), and is
|
|
found again by the ID, so renaming the team in the store does not lose it.
|
|
- **Import** works for quotas, by `team:ID` as `GET /api/v1/quotas` lists it
|
|
(the `scope` is then that ID; write it so in the configuration, or the next
|
|
plan replaces the quota with itself), for attachments, by
|
|
`scope:SCOPE-ID:policy` as their `id` shows, for organizations, teams,
|
|
projects and keys by their IDs (an imported key has no `secret`), and for
|
|
memberships by `USER-ID/scope:SCOPE-ID`. Models cannot be imported yet.
|
|
- **A project's `path` follows a move of its team at the next refresh**: the
|
|
project did not change, so the apply that moved the team does not rewrite
|
|
it.
|
|
|
|
## Developing it
|
|
|
|
A module of its own, so the gateway's binary does not carry Terraform's
|
|
dependencies; `go test ./...` at the repository's root does not include it.
|
|
|
|
```sh
|
|
cd terraform-provider-llmgw
|
|
go vet ./... && go test ./... # the client, against an admin API in memory and the OpenAPI document
|
|
TF_ACC=1 go test ./internal/provider/ # Terraform itself, plan and apply, against the same fake
|
|
```
|
|
|
|
- `internal/client` is the HTTP client. Its `Endpoints` lists every endpoint it
|
|
calls and it refuses any other; a test checks each of them, and every field
|
|
of every body it sends and reads, against the gateway's generated
|
|
[OpenAPI document](../docs/admin-api.openapi.json). A change to the admin API
|
|
that breaks the provider fails here.
|
|
- `internal/fakeapi` is an admin API in memory: the token, the content type,
|
|
the catalog's defaults and the tenancy tree's names, enough to run the
|
|
provider without a cluster.
|
|
- The acceptance tests need `TF_ACC=1` and a `terraform` on `PATH` (or
|
|
`TF_ACC_TERRAFORM_PATH`); without one, terraform-plugin-testing downloads the
|
|
latest. They pass with OpenTofu too:
|
|
`TF_ACC_TERRAFORM_PATH=$(which tofu) TF_ACC_PROVIDER_HOST=registry.opentofu.org`.
|