Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Ilker ManapandClaude Opus 5.5
|
99c34fd68f | ||
|
Ilker ManapandClaude Opus 5.5
|
cfc38d35cc | ||
|
Ilker ManapandClaude Opus 5.5
|
ac25a9136a | ||
|
Ilker ManapandClaude Opus 5.5
|
4b6b3eeddb | ||
|
Ilker ManapandClaude Opus 5.5
|
178b168bfd |
No files matched your search
+47
-1
@@ -18,12 +18,58 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## v0.14.9 — 2026-10-19
|
||||
|
||||
Sign-in set up from the console, with Active Directory tried before it is
|
||||
kept. A node's console now shows what the machine is doing, and a node's web
|
||||
page shows what it runs. Also fixes found trying Active Directory against a
|
||||
Samba domain controller on the test cluster. Sign-in through it, nested
|
||||
groups and a disabled account suspended by the sync all worked there.
|
||||
|
||||
**Upgrading.**
|
||||
- No migration.
|
||||
- `auth.ldap` and `auth.oidc` are in the cluster's settings, as before. A
|
||||
block written into `server.yaml` on a cluster with a store was never
|
||||
read; the log said so. Put it in with `llmgw ctl config edit`, or set it up
|
||||
again from the console's Sign-in screen.
|
||||
- A Samba DC's own autogenerated certificate is refused (a negative serial
|
||||
and no SAN). Give it one from a CA.
|
||||
|
||||
### Added
|
||||
|
||||
- **Sign-in set up from the console.** Under People → Sign-in, `sudo llmgw` shows how people get in: single sign-on,
|
||||
Active Directory and SCIM, each on or off. It also shows the directory's last sync. **E** sets up Entra ID from the
|
||||
tenant and client IDs. **D** sets up Active Directory, and the directory is tried before it is kept: each server
|
||||
reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
|
||||
password, client secret and SCIM token are kept in the store as `store:auth/…`, sealed by the master key. The
|
||||
domain's CA goes into the settings (`auth.ldap.ca`), so no file has to be put on each control plane. The admin API
|
||||
has the same: `GET /api/v1/auth`, `POST /api/v1/auth/ldap/test` and `PUT /api/v1/auth/secrets/{name}`.
|
||||
- **A node's page says what it runs.** It now shows the agent's version, marked when the control plane runs another,
|
||||
and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
|
||||
GPUs.
|
||||
- **`sudo llmgw` on a node opens on what the machine is doing.** A new Overview screen shows the agent's state and
|
||||
version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
|
||||
here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
|
||||
itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A broken `auth.ldap` no longer stops every control plane.** It is the cluster's setting. A CA file that one
|
||||
machine could not use made `llmgw server` exit on every control plane, and no `ctl config` could then put it right.
|
||||
Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
|
||||
- **A second `llmgw server` on a machine stops before it competes to lead.** `llmgw server check` (no such command)
|
||||
ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
|
||||
control planes stood by. `server` now refuses words it does not know and takes its ports before anything else.
|
||||
- **The single sign-on and Active Directory guides said `server.yaml`.** On a cluster with a store, `auth` is in the
|
||||
cluster's settings and the file's block is not read. They now say `llmgw ctl config edit`, and what certificate a
|
||||
domain controller needs.
|
||||
|
||||
## v0.14.8 — 2026-10-18
|
||||
|
||||
Sign-in with an on-premises Active Directory, and Entra ID made safe for
|
||||
|
||||
@@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
|
||||
@@ -23,6 +23,9 @@
|
||||
{
|
||||
"name": "audit"
|
||||
},
|
||||
{
|
||||
"name": "auth"
|
||||
},
|
||||
{
|
||||
"name": "canaries"
|
||||
},
|
||||
@@ -226,6 +229,129 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/auth": {
|
||||
"get": {
|
||||
"operationId": "getAuth",
|
||||
"summary": "How people sign in: OIDC, a directory, SCIM, and the directory's last sync",
|
||||
"tags": [
|
||||
"auth"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/AuthView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/auth/ldap/test": {
|
||||
"post": {
|
||||
"operationId": "postAuthLdapTest",
|
||||
"summary": "Try an auth.ldap block before it is kept: servers, the service account, people and groups",
|
||||
"tags": [
|
||||
"auth"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/LDAPTestRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/LDAPTestView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/auth/secrets/{name}": {
|
||||
"put": {
|
||||
"operationId": "putAuthSecretsName",
|
||||
"summary": "Keep ldap-bind, oidc-client-secret or scim-token in the store, for settings to name as store:auth/NAME",
|
||||
"tags": [
|
||||
"auth"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "ldap-bind, oidc-client-secret or scim-token.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SecretRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/canaries": {
|
||||
"get": {
|
||||
"operationId": "getCanaries",
|
||||
@@ -5351,6 +5477,31 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"AuthView": {
|
||||
"type": "object",
|
||||
"description": "AuthView is how people sign in on this gateway.",
|
||||
"properties": {
|
||||
"signup_open": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"oidc": {
|
||||
"$ref": "#/components/schemas/OIDCView",
|
||||
"description": "OIDC is single sign-on through an identity provider: Entra ID, Keycloak, Okta."
|
||||
},
|
||||
"ldap": {
|
||||
"$ref": "#/components/schemas/LDAPView",
|
||||
"description": "LDAP is sign-in checked by a directory: Active Directory."
|
||||
},
|
||||
"scim": {
|
||||
"type": "boolean",
|
||||
"description": "SCIM is whether a provider may keep accounts in step."
|
||||
},
|
||||
"base_url": {
|
||||
"type": "string",
|
||||
"description": "BaseURL is the portal's address, when the settings say it: what a provider's redirect address is made from."
|
||||
}
|
||||
}
|
||||
},
|
||||
"CanaryPercentRequest": {
|
||||
"type": "object",
|
||||
"description": "CanaryPercentRequest changes the share of a model's requests its canary takes.",
|
||||
@@ -5421,6 +5572,21 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"CheckView": {
|
||||
"type": "object",
|
||||
"description": "CheckView is one step of trying a directory.",
|
||||
"properties": {
|
||||
"step": {
|
||||
"type": "string"
|
||||
},
|
||||
"ok": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"detail": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"ClassGPUs": {
|
||||
"type": "object",
|
||||
"description": "ClassGPUs is how many of a node's devices are of one class, and how many of those are free or kept out of placement.",
|
||||
@@ -5573,6 +5739,34 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"DirectorySyncView": {
|
||||
"type": "object",
|
||||
"description": "DirectorySyncView is the last time the leader read the directory.",
|
||||
"properties": {
|
||||
"at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"people": {
|
||||
"type": "integer"
|
||||
},
|
||||
"accounts": {
|
||||
"type": "integer"
|
||||
},
|
||||
"suspended": {
|
||||
"type": "integer"
|
||||
},
|
||||
"reinstated": {
|
||||
"type": "integer"
|
||||
},
|
||||
"roles_changed": {
|
||||
"type": "integer"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"DomainView": {
|
||||
"type": "object",
|
||||
"description": "DomainView is one domain: its nodes, its power and whether it is held.",
|
||||
@@ -6826,6 +7020,61 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"LDAPTestRequest": {
|
||||
"type": "object",
|
||||
"description": "LDAPTestRequest is an auth.ldap block to try, as YAML, and the service account's password when it is not kept yet.",
|
||||
"properties": {
|
||||
"settings": {
|
||||
"type": "string"
|
||||
},
|
||||
"password": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"LDAPTestView": {
|
||||
"type": "object",
|
||||
"description": "LDAPTestView is how trying the directory went.",
|
||||
"properties": {
|
||||
"ok": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"checks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/CheckView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"LDAPView": {
|
||||
"type": "object",
|
||||
"description": "LDAPView is auth.ldap as it is in force, and how its last sync went.",
|
||||
"properties": {
|
||||
"urls": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"base_dn": {
|
||||
"type": "string"
|
||||
},
|
||||
"bind_dn": {
|
||||
"type": "string"
|
||||
},
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"off": {
|
||||
"type": "string",
|
||||
"description": "Off is why this control plane could not use the settings, when it could not; local accounts still sign in."
|
||||
},
|
||||
"last_sync": {
|
||||
"$ref": "#/components/schemas/DirectorySyncView"
|
||||
}
|
||||
}
|
||||
},
|
||||
"Live": {
|
||||
"type": "object",
|
||||
"description": "Live is one of the cluster's adapters as its node last reported it.",
|
||||
@@ -7492,6 +7741,27 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"OIDCView": {
|
||||
"type": "object",
|
||||
"description": "OIDCView is auth.oidc as it is in force.",
|
||||
"properties": {
|
||||
"issuer": {
|
||||
"type": "string"
|
||||
},
|
||||
"client_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"create_accounts": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"no_passwords": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"OrgView": {
|
||||
"type": "object",
|
||||
"description": "OrgView is an organization with what is in it.",
|
||||
@@ -8182,6 +8452,15 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"SecretRequest": {
|
||||
"type": "object",
|
||||
"description": "SecretRequest is a secret for the store to keep.",
|
||||
"properties": {
|
||||
"value": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"SettingsRequest": {
|
||||
"type": "object",
|
||||
"description": "SettingsRequest is a new version of the cluster's settings, made from the version Base, which must still be the newest.",
|
||||
|
||||
+39
-4
@@ -2,8 +2,8 @@
|
||||
|
||||
`sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
|
||||
with nothing installed it opens the setup screens; on a control plane, the
|
||||
console described here; on a node, the node's own console, which edits
|
||||
`agent.yaml`. Everything on these screens can also be done by command — each
|
||||
console described here; on a node, the node's own console, which shows
|
||||
what runs on that machine and edits `agent.yaml`. Everything on these screens can also be done by command — each
|
||||
screen says which — so a script never needs them and a person never needs
|
||||
to remember the commands.
|
||||
|
||||
@@ -246,6 +246,28 @@ long a key made there may live, and is the expiry of one made without any
|
||||
Limits are requests and tokens a minute, a budget per day or month, and the
|
||||
models allowed. Each level only narrows what is above it.
|
||||
|
||||
**Sign-in** — how people get in besides a password kept here: single
|
||||
sign-on, Active Directory and provisioning, each on or off, with the
|
||||
directory's last sync (how many people, accounts kept in step, suspended,
|
||||
reinstated).
|
||||
- **E** sets up Entra ID. Give the tenant ID, the client ID and secret,
|
||||
and the groups for administrators and operators. A provider other than
|
||||
Entra takes its issuer URL in place of the tenant. The form shows the
|
||||
redirect address to register.
|
||||
- **D** sets up Active Directory: the servers, where people are, the
|
||||
service account and its password, the domain's CA certificate as a PEM
|
||||
file on this machine, the group whose members may sign in, and groups
|
||||
for roles and teams.
|
||||
- Before it is kept, the directory is tried from the control plane: each
|
||||
server reached and bound, how many people may sign in, how many are in
|
||||
each group. If any step fails, nothing is kept and the steps are shown.
|
||||
**T** tries the settings in force.
|
||||
- On the provisioning row, Enter makes a SCIM token, shown once.
|
||||
- The password, the client secret and the token are kept in the store,
|
||||
sealed by the master key when one is set. The settings name them as
|
||||
`store:auth/…`, so each control plane has them with no file put on it.
|
||||
The CA is written into the settings themselves.
|
||||
|
||||
**Guardrails…** on a row lists the guardrail policies server.yaml defines.
|
||||
Each policy is marked in one of three ways:
|
||||
|
||||
@@ -335,5 +357,18 @@ to restart it.
|
||||
|
||||
## The node's console
|
||||
|
||||
On a node, `sudo llmgw` opens `agent.yaml` in the same settings view, and the
|
||||
manual. The cluster is watched and changed from a control plane.
|
||||
On a node, `sudo llmgw` opens on **Overview**: what this machine is doing, read
|
||||
on the machine itself, so it works without an admin token and while the control
|
||||
plane is out of reach. Refreshed every three seconds, it shows
|
||||
|
||||
- llmgw-agent's state and since when, and the version of llmgw;
|
||||
- each control plane the agent dials, from `agent.yaml` and the list the
|
||||
leader last sent, and whether it can be reached;
|
||||
- each GPU: how busy, its memory, its heat, and the models on it;
|
||||
- the engines running here, from the agent's own records: replica, model,
|
||||
engine, GPUs, port, whether the process is still there, and for how long;
|
||||
- the free space where weights and engines are kept;
|
||||
- the agent's warnings and errors of the last hour.
|
||||
|
||||
Below it, `agent.yaml` in the same settings view, and the manual. The cluster
|
||||
is watched and changed from a control plane.
|
||||
+22
-3
@@ -2491,8 +2491,10 @@ Google, Okta, Authentik, anything that speaks OpenID Connect.
|
||||
|
||||
1. Register a client there with the redirect address
|
||||
`https://<gateway>/login/oidc/callback`.
|
||||
2. Put its secret in a file only root reads.
|
||||
3. Add to `server.yaml`:
|
||||
2. Put its secret in a file the llmgw user reads, on every control plane.
|
||||
3. Add this to the cluster's settings with `llmgw ctl config edit`. On a
|
||||
cluster with a store, `auth` is the cluster's: the same block in
|
||||
`server.yaml` is not read, and the log says so.
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
@@ -2543,6 +2545,9 @@ and point it at `https://<gateway>/scim/v2` with that token as the bearer.
|
||||
|
||||
### Entra ID
|
||||
|
||||
The console sets this up: `sudo llmgw`, People → Sign-in, **E**. What
|
||||
follows is what it writes, and what to do in Entra.
|
||||
|
||||
Entra ID is an OpenID Connect provider like the others, with two things to
|
||||
set in the app registration:
|
||||
|
||||
@@ -2568,7 +2573,14 @@ the secret token to the contents of `auth.scim.token_file`.
|
||||
|
||||
With an on-premises Active Directory, people sign in on the gateway's own
|
||||
form with their AD username (`ada`, `CORP\ada`, `ada@corp.example`) and
|
||||
password. The gateway checks them with the domain controllers over LDAPS:
|
||||
password. The gateway checks them with the domain controllers over LDAPS.
|
||||
|
||||
The console sets this up and tries it first: `sudo llmgw`, People →
|
||||
Sign-in, **D**. It keeps the service account's password in the store
|
||||
(`bind_password_file: store:auth/ldap-bind`) and writes the CA into the
|
||||
settings (`ca:`), so nothing has to be copied to each control plane.
|
||||
By hand it is this block in the cluster's settings (`llmgw ctl config
|
||||
edit`), where files it names must then be on every control plane:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
@@ -2595,6 +2607,13 @@ auth:
|
||||
checked by binding as them, and is never stored.
|
||||
- **Encryption:** `ldaps://`, or `ldap://` with `start_tls: true`.
|
||||
Plaintext LDAP is refused, since passwords would cross the network.
|
||||
- **Certificate:** the domain controller's certificate must name it in
|
||||
its subject alternative names: a name in the CN only is refused, as are
|
||||
certificates with a negative serial number. Samba's own autogenerated
|
||||
certificate is both, so a Samba DC needs one from a CA (`tls certfile`
|
||||
in smb.conf). A CA file a control plane cannot read turns directory
|
||||
sign-in off on that control plane, said in its log. Local accounts still
|
||||
sign in.
|
||||
- **Who may sign in:** enabled user objects under `base_dn` that match
|
||||
`user_filter`. The example admits members of `llmgw-users`, nested groups
|
||||
included.
|
||||
|
||||
@@ -151,6 +151,7 @@ rather than repeated.
|
||||
| `urls` | `[]string` | [LDAP](#ldap) |
|
||||
| `start_tls` | `bool` | [LDAP](#ldap) |
|
||||
| `ca_file` | `string` | [LDAP](#ldap) |
|
||||
| `ca` | `string` | [LDAP](#ldap) |
|
||||
| `bind_dn` | `string` | [LDAP](#ldap) |
|
||||
| `bind_password_file` | `string` | [LDAP](#ldap) |
|
||||
| `base_dn` | `string` | [LDAP](#ldap) |
|
||||
@@ -1475,6 +1476,14 @@ is refused.
|
||||
CAFile is the authority the servers' certificates are checked against,
|
||||
when it is not the system's: an AD Certificate Services root.
|
||||
|
||||
### `ca`
|
||||
|
||||
`string`
|
||||
|
||||
CA is the same authority written out, PEM: kept in the cluster's
|
||||
settings, so no file has to be put on every control plane. Used in
|
||||
place of CAFile when both are set.
|
||||
|
||||
### `bind_dn`
|
||||
|
||||
`string`
|
||||
|
||||
@@ -74,6 +74,13 @@ func NewSensorSampler() *SensorSampler {
|
||||
return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
|
||||
}
|
||||
|
||||
// SampleGPUs reads every card's heat, power and how busy it is now, without
|
||||
// a sampler's ten seconds of memory: for a screen that asks every few seconds
|
||||
// and wants what it asked for. Nil on a machine without nvidia-smi or a card.
|
||||
func SampleGPUs(ctx context.Context) []GPUSensor {
|
||||
return (&SensorSampler{smi: runSMI}).gpus(ctx)
|
||||
}
|
||||
|
||||
func runSMI(ctx context.Context, fields string) ([]byte, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
+22
-15
@@ -76,6 +76,7 @@ func newServerCmd() *cobra.Command {
|
||||
|
||||
cmd := &cobra.Command{
|
||||
Use: "server",
|
||||
Args: cobra.NoArgs,
|
||||
Short: "Run the router (phase 1; the control plane arrives in phase 3)",
|
||||
Long: "Serves the OpenAI API in front of a set of engine replicas.\n\n" +
|
||||
"In this phase the replica list is static: give it in the config file, or\n" +
|
||||
@@ -138,6 +139,26 @@ func newServerCmd() *cobra.Command {
|
||||
_ = shutdownTracing(sctx)
|
||||
}()
|
||||
|
||||
// The ports before anything else: a second server on this machine,
|
||||
// whose ports are taken, must stop here, not go on to compete for
|
||||
// the lease and lead a cluster no request can reach.
|
||||
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||
}
|
||||
if cfg.MaxConnections > 0 {
|
||||
// Past this, new connections wait in the kernel's backlog
|
||||
// rather than each taking a goroutine and its buffers.
|
||||
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
||||
}
|
||||
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||
if err != nil {
|
||||
_ = apiLn.Close()
|
||||
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
||||
}
|
||||
defer apiLn.Close()
|
||||
defer adminLn.Close()
|
||||
|
||||
rt, err := router.New(cfg, log)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -418,6 +439,7 @@ func newServerCmd() *cobra.Command {
|
||||
}
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
api.SetAuth(&adminapi.AuthAdmin{Status: rt.AuthStatus, TestLDAP: rt.TestDirectory, Secrets: st, Log: log})
|
||||
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
||||
Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
|
||||
// And the same document on the portal, for administrators.
|
||||
@@ -661,21 +683,6 @@ func newServerCmd() *cobra.Command {
|
||||
ReadTimeout: time.Minute,
|
||||
})
|
||||
|
||||
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||
}
|
||||
if cfg.MaxConnections > 0 {
|
||||
// Past this, new connections wait in the kernel's backlog
|
||||
// rather than each taking a goroutine and its buffers.
|
||||
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
||||
}
|
||||
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||
if err != nil {
|
||||
_ = apiLn.Close()
|
||||
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
||||
}
|
||||
|
||||
// A metrics-only listener, when one is configured. Started after
|
||||
// AdminHandler, which is where the registry it serves is built.
|
||||
var metricsSrv *http.Server
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// `llmgw server check` once ran a whole second server: words after server
|
||||
// that name no subcommand are refused.
|
||||
func TestServerTakesNoWordsItDoesNotKnow(t *testing.T) {
|
||||
cmd := newServerCmd()
|
||||
cmd.SetArgs([]string{"check", "--config", "/nonexistent"})
|
||||
cmd.SilenceUsage, cmd.SilenceErrors = true, true
|
||||
if err := cmd.Execute(); err == nil || !strings.Contains(err.Error(), "unknown command") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A second server whose port is taken stops at once, before the router or
|
||||
// any control plane is built.
|
||||
func TestAServerWhosePortIsTakenStopsFirst(t *testing.T) {
|
||||
taken, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer taken.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
cmd := newServerCmd()
|
||||
cmd.SetArgs([]string{"--listen", taken.Addr().String(), "--admin-listen", "127.0.0.1:0",
|
||||
"--endpoint", "a=http://127.0.0.1:1=m"})
|
||||
cmd.SilenceUsage, cmd.SilenceErrors = true, true
|
||||
if err := cmd.ExecuteContext(ctx); err == nil || !strings.Contains(err.Error(), "listening on") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
}
|
||||
+44
-15
@@ -53,6 +53,7 @@ const (
|
||||
cAccounts
|
||||
cKeys
|
||||
cOrgs
|
||||
cSignIn
|
||||
cAudit
|
||||
cManual
|
||||
cSettings
|
||||
@@ -69,7 +70,7 @@ const (
|
||||
cCount
|
||||
)
|
||||
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
|
||||
|
||||
@@ -140,8 +141,8 @@ type consoleModel struct {
|
||||
detail bool
|
||||
scroll int
|
||||
|
||||
// node is the console of a node: its agent.yaml and nothing to watch,
|
||||
// since the cluster is watched from the control plane.
|
||||
// node is the console of a node: what runs on the machine and its
|
||||
// agent.yaml, since the cluster is watched from the control plane.
|
||||
node bool
|
||||
|
||||
// The settings.
|
||||
@@ -181,6 +182,7 @@ type consoleModel struct {
|
||||
quotas []adminapi.QuotaView
|
||||
quotasErr error
|
||||
guard guardState
|
||||
signIn signInState
|
||||
kb kbState
|
||||
|
||||
st *storeHolder
|
||||
@@ -190,6 +192,11 @@ type consoleModel struct {
|
||||
docs docViewer
|
||||
mon monitorState
|
||||
det monitorState
|
||||
|
||||
// nodeOv is a node's overview as last read; nodeBusy says a reading is
|
||||
// under way.
|
||||
nodeOv nodeOverview
|
||||
nodeBusy bool
|
||||
}
|
||||
|
||||
var usageGroupings = []string{"team", "project", "org"}
|
||||
@@ -198,9 +205,6 @@ func newConsoleModel(addr, file string) consoleModel {
|
||||
m := consoleModel{client: newCtlClient(addr), spin: newSpinner(), width: 100, height: 30, file: file,
|
||||
st: &storeHolder{file: file}, docs: newDocViewer()}
|
||||
m.node = strings.HasSuffix(file, "agent.yaml")
|
||||
if m.node {
|
||||
m.menu = cSettings
|
||||
}
|
||||
m.loadSettings()
|
||||
return m
|
||||
}
|
||||
@@ -257,7 +261,7 @@ func (m *consoleModel) loadSettings() {
|
||||
|
||||
func (m consoleModel) Init() tea.Cmd {
|
||||
if m.node {
|
||||
return m.spin.Tick
|
||||
return tea.Batch(m.fetchNodeOverview(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
|
||||
}
|
||||
return tea.Batch(m.fetch(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
|
||||
}
|
||||
@@ -320,7 +324,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
return m, cmd
|
||||
case consoleTickMsg:
|
||||
if m.node {
|
||||
return m, nil
|
||||
return m.nodeTick()
|
||||
}
|
||||
cmds := []tea.Cmd{m.fetch(), tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })}
|
||||
if m.menu == cUsage {
|
||||
@@ -332,6 +336,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
if m.menu == cGuard {
|
||||
cmds = append(cmds, m.fetchGuard())
|
||||
}
|
||||
if m.menu == cSignIn {
|
||||
cmds = append(cmds, m.fetchSignIn())
|
||||
}
|
||||
if m.menu == cKB {
|
||||
cmds = append(cmds, m.fetchKB())
|
||||
}
|
||||
@@ -351,6 +358,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
cmds = append(cmds, m.fetchLogs(m.logs.node))
|
||||
}
|
||||
return m, tea.Batch(cmds...)
|
||||
case nodeOverviewMsg:
|
||||
m.nodeOv, m.nodeBusy = nodeOverview(msg), false
|
||||
case consoleDataMsg:
|
||||
m.loaded = true
|
||||
if msg.err != nil {
|
||||
@@ -395,6 +404,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
m.quotas, m.quotasErr = msg.rows, msg.err
|
||||
case consoleKBMsg:
|
||||
m.kb = kbState{view: msg.view, err: msg.err, loaded: true}
|
||||
case consoleSignInMsg:
|
||||
m.signIn = signInState{view: msg.view, err: msg.err, loaded: true, ldap: msg.ldap, oidc: msg.oidc}
|
||||
case consoleGuardMsg:
|
||||
window := m.guard.window
|
||||
m.guard = guardState{view: msg.view, events: msg.events, err: msg.err, window: window}
|
||||
@@ -423,7 +434,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
|
||||
func (m consoleModel) entryOK(i int) bool {
|
||||
if m.node {
|
||||
return i == cSettings || i == cManual || i == cWipe || i == cLeave
|
||||
return i == cOverview || i == cSettings || i == cManual || i == cWipe || i == cLeave
|
||||
}
|
||||
return i != cReplicate || !m.storeReplicated()
|
||||
}
|
||||
@@ -553,6 +564,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.menuD = m.quotaMenu(m.quotas[m.row])
|
||||
case cGuard:
|
||||
m.menuD = m.guardAttachmentMenu(m.guard.view.Attachments[m.row])
|
||||
case cSignIn:
|
||||
m.menuD = m.signInMenu(m.signInRowKey())
|
||||
case cKB:
|
||||
m.menuD = m.kbMenu(m.kb.view.Collections[m.row])
|
||||
case cMirror:
|
||||
@@ -649,6 +662,19 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.usageExportForm()
|
||||
return m, nil
|
||||
}
|
||||
if m.menu == cSignIn && m.signIn.loaded {
|
||||
m.entraForm()
|
||||
return m, nil
|
||||
}
|
||||
case "d":
|
||||
if m.menu == cSignIn && m.signIn.loaded {
|
||||
m.directoryForm()
|
||||
return m, nil
|
||||
}
|
||||
case "t":
|
||||
if m.menu == cSignIn && m.signIn.view.LDAP != nil {
|
||||
return m, m.tryDirectory()
|
||||
}
|
||||
case "right", "l":
|
||||
if m.menu == cUsage {
|
||||
m.usageBy = (m.usageBy + 1) % len(usageGroupings)
|
||||
@@ -675,6 +701,8 @@ func (m consoleModel) onEnter() tea.Cmd {
|
||||
return m.fetchQuotas()
|
||||
case cGuard:
|
||||
return m.fetchGuard()
|
||||
case cSignIn:
|
||||
return m.fetchSignIn()
|
||||
case cKB:
|
||||
return m.fetchKB()
|
||||
case cMirror:
|
||||
@@ -1053,6 +1081,8 @@ func (m consoleModel) rowCount() int {
|
||||
return len(m.quotas)
|
||||
case cGuard:
|
||||
return len(m.guard.view.Attachments)
|
||||
case cSignIn:
|
||||
return len(signInRows)
|
||||
case cKB:
|
||||
return len(m.kb.view.Collections)
|
||||
case cMirror:
|
||||
@@ -1080,9 +1110,6 @@ func (m consoleModel) rowCount() int {
|
||||
func (m consoleModel) View() string {
|
||||
host, _ := os.Hostname()
|
||||
nav := []navItem{{Label: "Watch", Heading: true}}
|
||||
if m.node {
|
||||
nav = nil
|
||||
}
|
||||
for i := 0; i < cCount; i++ {
|
||||
if m.node && !m.entryOK(i) {
|
||||
continue
|
||||
@@ -1177,6 +1204,9 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
}
|
||||
switch m.menu {
|
||||
case cOverview:
|
||||
if m.node {
|
||||
return m.nodeOverviewPanel(pw)
|
||||
}
|
||||
var b bytes.Buffer
|
||||
c := &cobra.Command{}
|
||||
c.SetOut(&b)
|
||||
@@ -1292,6 +1322,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
return m.quotasPanel(pw)
|
||||
case cGuard:
|
||||
return m.guardPanel(pw)
|
||||
case cSignIn:
|
||||
return m.signInPanel(pw)
|
||||
case cKB:
|
||||
return m.kbPanel(pw)
|
||||
case cAccounts, cKeys, cOrgs:
|
||||
@@ -1648,9 +1680,6 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
|
||||
}
|
||||
}
|
||||
menu := cOverview
|
||||
if strings.HasSuffix(file, "agent.yaml") {
|
||||
menu = cSettings
|
||||
}
|
||||
for {
|
||||
m := newConsoleModel(addr, file)
|
||||
m.menu = menu
|
||||
|
||||
@@ -0,0 +1,715 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"github.com/charmbracelet/lipgloss"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
|
||||
// A node's overview: what a bare `sudo llmgw` opens on a machine that runs
|
||||
// only llmgw-agent. Everything on it is read here — the agent's own files,
|
||||
// systemd, the cards and the disks — because a node has no admin token, and
|
||||
// the control plane may be the very thing it cannot reach.
|
||||
|
||||
// nodeCmdTimeout bounds every command and check the overview runs: a screen
|
||||
// refreshed every three seconds cannot wait on a tool that hangs.
|
||||
const nodeCmdTimeout = 2 * time.Second
|
||||
|
||||
// nodeControlPlanesFile is where the agent keeps the control planes the
|
||||
// leader last named (controlPlanesFile in the agent package).
|
||||
var nodeControlPlanesFile = "/var/lib/llmgw-agent/control-planes.json"
|
||||
|
||||
// nodeProblemLines is how many of the agent's warnings the overview shows.
|
||||
const nodeProblemLines = 6
|
||||
|
||||
type nodeOverviewMsg nodeOverview
|
||||
|
||||
// nodeOverview is everything the overview shows, read in one go.
|
||||
type nodeOverview struct {
|
||||
at time.Time
|
||||
version string
|
||||
agent agentService
|
||||
planes []planeCheck
|
||||
gpus []nodeGPU
|
||||
// gpuNote says why there are no GPU rows: no card, or no answer.
|
||||
gpuNote string
|
||||
// names is the cards' names by index, which do not change and are asked
|
||||
// for once rather than every refresh.
|
||||
names map[int]string
|
||||
engines []engineRun
|
||||
enginesNote string
|
||||
disks []diskUse
|
||||
problems []string
|
||||
// problemsNote replaces the problems when the journal cannot be read.
|
||||
problemsNote string
|
||||
}
|
||||
|
||||
// agentService is what systemd says of llmgw-agent.
|
||||
type agentService struct {
|
||||
state, sub string
|
||||
since time.Time
|
||||
// note replaces the rest when systemd cannot say.
|
||||
note string
|
||||
}
|
||||
|
||||
// planeCheck is one control plane the agent dials, and whether it answers.
|
||||
type planeCheck struct {
|
||||
addr string
|
||||
// learned is an address the leader named rather than one in agent.yaml.
|
||||
learned bool
|
||||
err string
|
||||
}
|
||||
|
||||
type nodeGPU struct {
|
||||
index int
|
||||
name string
|
||||
// util is -1 where the driver does not report it.
|
||||
util float64
|
||||
memUsedMB, memTotalMB float64
|
||||
tempC float64
|
||||
models []string
|
||||
}
|
||||
|
||||
// engineRun is one engine the supervisor started here, from its state file.
|
||||
type engineRun struct {
|
||||
replica, model, engine string
|
||||
gpus []int
|
||||
port, pid int
|
||||
alive bool
|
||||
started time.Time
|
||||
}
|
||||
|
||||
type diskUse struct {
|
||||
what, path string
|
||||
free, total uint64
|
||||
err string
|
||||
}
|
||||
|
||||
// nodeProbe reads this machine. Every path and tool is a field, so a test
|
||||
// reads a directory of its own instead.
|
||||
type nodeProbe struct {
|
||||
configured []string
|
||||
planesFile string
|
||||
workDir string
|
||||
weightsDir string
|
||||
enginesDir string
|
||||
root bool
|
||||
names map[int]string
|
||||
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
dial func(ctx context.Context, addr string) error
|
||||
gpus func(ctx context.Context) []discovery.GPUSensor
|
||||
cards func(ctx context.Context) ([]discovery.GPU, error)
|
||||
alive func(pid int) bool
|
||||
statfs func(path string) (free, total uint64, err error)
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
// newNodeProbe reads the machine agent.yaml describes, as the agent would.
|
||||
func newNodeProbe(cfg any, names map[int]string) nodeProbe {
|
||||
c, ok := cfg.(agent.Config)
|
||||
if !ok {
|
||||
c = agent.DefaultConfig()
|
||||
}
|
||||
var configured []string
|
||||
if c.Server != "" {
|
||||
configured = append(configured, c.Server)
|
||||
}
|
||||
configured = append(configured, c.Servers...)
|
||||
// The same defaults the agent applies as it starts.
|
||||
weights := c.WeightsCache
|
||||
if weights == "" && len(c.ModelDirs) > 0 {
|
||||
weights = c.ModelDirs[0]
|
||||
}
|
||||
if weights == "" {
|
||||
weights = agent.DefaultWeightsCache
|
||||
}
|
||||
enginesDir := c.EnginesDir
|
||||
if enginesDir == "" {
|
||||
enginesDir = provision.DefaultRoot
|
||||
}
|
||||
work := c.Superv.WorkDir
|
||||
if work == "" {
|
||||
work = "/var/lib/llmgw-agent"
|
||||
}
|
||||
return nodeProbe{
|
||||
configured: configured, planesFile: nodeControlPlanesFile, workDir: work,
|
||||
weightsDir: weights, enginesDir: enginesDir, root: os.Geteuid() == 0, names: names,
|
||||
run: runLocal, dial: dialTCP, gpus: discovery.SampleGPUs,
|
||||
cards: func(ctx context.Context) ([]discovery.GPU, error) { return discovery.DiscoverGPUsSMI(ctx, "") },
|
||||
alive: pidAlive, statfs: statfs, now: time.Now,
|
||||
}
|
||||
}
|
||||
|
||||
// fetchNodeOverview reads the machine in the background.
|
||||
func (m consoleModel) fetchNodeOverview() tea.Cmd {
|
||||
p := newNodeProbe(m.cfg, m.nodeOv.names)
|
||||
return func() tea.Msg { return nodeOverviewMsg(p.gather(context.Background())) }
|
||||
}
|
||||
|
||||
// nodeTick refreshes the overview while it is on screen, one reading at a
|
||||
// time; the other screens of a node watch nothing.
|
||||
func (m consoleModel) nodeTick() (consoleModel, tea.Cmd) {
|
||||
tick := tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })
|
||||
if m.menu != cOverview || m.nodeBusy {
|
||||
return m, tick
|
||||
}
|
||||
m.nodeBusy = true
|
||||
return m, tea.Batch(m.fetchNodeOverview(), tick)
|
||||
}
|
||||
|
||||
// gather reads everything at once, each part in its own goroutine, so the
|
||||
// slowest part bounds the whole.
|
||||
func (p nodeProbe) gather(ctx context.Context) nodeOverview {
|
||||
ov := nodeOverview{version: version.Version}
|
||||
var wg sync.WaitGroup
|
||||
do := func(f func()) {
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); f() }()
|
||||
}
|
||||
var engines []engineRun
|
||||
do(func() { ov.agent = p.agentState(ctx) })
|
||||
do(func() { ov.planes = p.checkPlanes(ctx) })
|
||||
do(func() { ov.gpus, ov.names, ov.gpuNote = p.readGPUs(ctx) })
|
||||
do(func() { engines, ov.enginesNote = readShardStates(p.workDir, p.alive) })
|
||||
do(func() { ov.disks = p.readDisks() })
|
||||
do(func() { ov.problems, ov.problemsNote = p.readProblems(ctx) })
|
||||
wg.Wait()
|
||||
ov.engines = engines
|
||||
ov.gpus = modelsOnGPUs(ov.gpus, engines)
|
||||
ov.at = p.now()
|
||||
return ov
|
||||
}
|
||||
|
||||
// --- the agent ---
|
||||
|
||||
func (p nodeProbe) agentState(ctx context.Context) agentService {
|
||||
out, err := p.run(ctx, "systemctl", "show", "llmgw-agent",
|
||||
"-p", "LoadState", "-p", "ActiveState", "-p", "SubState", "-p", "ActiveEnterTimestamp")
|
||||
if err != nil && len(out) == 0 {
|
||||
return agentService{note: "systemctl is not available: " + err.Error()}
|
||||
}
|
||||
return parseSystemctlShow(string(out))
|
||||
}
|
||||
|
||||
// parseSystemctlShow reads `systemctl show -p …`: one key=value a line.
|
||||
func parseSystemctlShow(out string) agentService {
|
||||
kv := map[string]string{}
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
|
||||
kv[k] = v
|
||||
}
|
||||
}
|
||||
if kv["LoadState"] == "not-found" {
|
||||
return agentService{note: "llmgw-agent is not installed as a service here"}
|
||||
}
|
||||
s := agentService{state: kv["ActiveState"], sub: kv["SubState"]}
|
||||
if s.state == "" {
|
||||
s.note = "systemd does not say"
|
||||
}
|
||||
s.since = parseSystemdTime(kv["ActiveEnterTimestamp"])
|
||||
return s
|
||||
}
|
||||
|
||||
// parseSystemdTime reads systemd's "Mon 2026-10-19 08:12:03 CEST", in this
|
||||
// machine's zone; zero when it is empty or n/a.
|
||||
func parseSystemdTime(v string) time.Time {
|
||||
v = strings.TrimSpace(v)
|
||||
if v == "" || v == "n/a" {
|
||||
return time.Time{}
|
||||
}
|
||||
for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05 -0700", "Mon 2006-01-02 15:04:05"} {
|
||||
if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
// --- the control planes ---
|
||||
|
||||
// readControlPlanes is every address the agent dials, in its order: the
|
||||
// configured ones, then those the leader named that agent.yaml lacks.
|
||||
func readControlPlanes(file string, configured []string) []planeCheck {
|
||||
var out []planeCheck
|
||||
seen := map[string]bool{}
|
||||
add := func(s string, learned bool) {
|
||||
hp := hostPort(s)
|
||||
if hp == "" || seen[hp] {
|
||||
return
|
||||
}
|
||||
seen[hp] = true
|
||||
out = append(out, planeCheck{addr: hp, learned: learned})
|
||||
}
|
||||
for _, s := range configured {
|
||||
add(s, false)
|
||||
}
|
||||
if b, err := os.ReadFile(file); err == nil {
|
||||
var told []string
|
||||
if json.Unmarshal(b, &told) == nil {
|
||||
for _, s := range told {
|
||||
add(s, true)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// hostPort is the host:port a control plane's address dials, whether it was
|
||||
// written as a URL or not.
|
||||
func hostPort(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
port := ""
|
||||
switch {
|
||||
case strings.HasPrefix(s, "https://"):
|
||||
s, port = strings.TrimPrefix(s, "https://"), "443"
|
||||
case strings.HasPrefix(s, "http://"):
|
||||
s, port = strings.TrimPrefix(s, "http://"), "80"
|
||||
}
|
||||
s, _, _ = strings.Cut(s, "/")
|
||||
if _, _, err := net.SplitHostPort(s); err != nil && port != "" && s != "" {
|
||||
s = net.JoinHostPort(strings.Trim(s, "[]"), port)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func (p nodeProbe) checkPlanes(ctx context.Context) []planeCheck {
|
||||
planes := readControlPlanes(p.planesFile, p.configured)
|
||||
var wg sync.WaitGroup
|
||||
for i := range planes {
|
||||
wg.Add(1)
|
||||
go func(c *planeCheck) {
|
||||
defer wg.Done()
|
||||
if err := p.dial(ctx, c.addr); err != nil {
|
||||
c.err = err.Error()
|
||||
}
|
||||
}(&planes[i])
|
||||
}
|
||||
wg.Wait()
|
||||
return planes
|
||||
}
|
||||
|
||||
func dialTCP(ctx context.Context, addr string) error {
|
||||
d := net.Dialer{Timeout: nodeCmdTimeout}
|
||||
c, err := d.DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return c.Close()
|
||||
}
|
||||
|
||||
// --- the GPUs ---
|
||||
|
||||
// readGPUs samples the cards, naming them from discovery the first time.
|
||||
func (p nodeProbe) readGPUs(ctx context.Context) ([]nodeGPU, map[int]string, string) {
|
||||
ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
|
||||
defer cancel()
|
||||
sensors := p.gpus(ctx)
|
||||
if len(sensors) == 0 {
|
||||
// Without nvidia-smi discovery says nothing and no error: a machine
|
||||
// without a card. With it and no answer, a driver in trouble.
|
||||
if _, err := p.cards(ctx); err != nil {
|
||||
return nil, p.names, "The GPUs do not answer: " + err.Error()
|
||||
}
|
||||
return nil, p.names, "No GPU on this machine."
|
||||
}
|
||||
names := p.names
|
||||
for _, s := range sensors {
|
||||
if names[s.Index] == "" {
|
||||
names = map[int]string{}
|
||||
if cards, err := p.cards(ctx); err == nil {
|
||||
for _, c := range cards {
|
||||
if c.MIG == nil {
|
||||
names[c.Index] = c.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
var out []nodeGPU
|
||||
for _, s := range sensors {
|
||||
g := nodeGPU{index: s.Index, name: names[s.Index], util: s.UtilPercent, tempC: s.TempC,
|
||||
memUsedMB: s.MemoryUsedMB, memTotalMB: s.MemoryTotalMB}
|
||||
if s.MemoryTotalMB == 0 {
|
||||
g.util = -1
|
||||
}
|
||||
out = append(out, g)
|
||||
}
|
||||
return out, names, ""
|
||||
}
|
||||
|
||||
// cardOf is the card a device number is on: itself, or a MIG instance's card.
|
||||
func cardOf(i int) int {
|
||||
if discovery.IsMIGIndex(i) {
|
||||
return (i - discovery.MIGIndex(0, 0)) / 100
|
||||
}
|
||||
return i
|
||||
}
|
||||
|
||||
// modelsOnGPUs names, on each card, the models whose engines hold it.
|
||||
func modelsOnGPUs(gpus []nodeGPU, runs []engineRun) []nodeGPU {
|
||||
for i := range gpus {
|
||||
gpus[i].models = nil
|
||||
for _, r := range runs {
|
||||
for _, d := range r.gpus {
|
||||
if cardOf(d) == gpus[i].index && !slices.Contains(gpus[i].models, r.model) {
|
||||
gpus[i].models = append(gpus[i].models, r.model)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return gpus
|
||||
}
|
||||
|
||||
// --- the engines ---
|
||||
|
||||
// shardStateFile is the part of the supervisor's shard-*.json the overview
|
||||
// reads (shardState in the supervisor package).
|
||||
type shardStateFile struct {
|
||||
ShardID string `json:"shard_id"`
|
||||
ReplicaID string `json:"replica_id"`
|
||||
PID int `json:"pid"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
Spec engines.ShardSpec `json:"spec"`
|
||||
}
|
||||
|
||||
// readShardStates is the engines the supervisor keeps state files for in
|
||||
// its work directory, with whether each process is still there.
|
||||
func readShardStates(dir string, alive func(int) bool) ([]engineRun, string) {
|
||||
entries, err := os.ReadDir(dir)
|
||||
switch {
|
||||
case errors.Is(err, os.ErrPermission):
|
||||
return nil, "The engines' state is root's: run with sudo."
|
||||
case errors.Is(err, os.ErrNotExist):
|
||||
return nil, ""
|
||||
case err != nil:
|
||||
return nil, err.Error()
|
||||
}
|
||||
var runs []engineRun
|
||||
note := ""
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if !strings.HasPrefix(name, "shard-") || !strings.HasSuffix(name, ".json") {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, name))
|
||||
if errors.Is(err, os.ErrPermission) {
|
||||
note = "The engines' state is root's: run with sudo."
|
||||
continue
|
||||
}
|
||||
var st shardStateFile
|
||||
if err != nil || json.Unmarshal(b, &st) != nil {
|
||||
continue
|
||||
}
|
||||
replica := st.ReplicaID
|
||||
if replica == "" {
|
||||
replica = st.Spec.ReplicaID
|
||||
}
|
||||
model := st.Spec.Profile.ServedModelName
|
||||
if model == "" {
|
||||
model = modelOfReplica(replica)
|
||||
}
|
||||
runs = append(runs, engineRun{replica: replica, model: model, engine: st.Spec.Profile.Engine,
|
||||
gpus: st.Spec.GPUIndices, port: st.Spec.HTTPPort, pid: st.PID, alive: st.PID > 0 && alive(st.PID),
|
||||
started: st.StartedAt})
|
||||
}
|
||||
sort.Slice(runs, func(i, j int) bool { return runs[i].replica < runs[j].replica })
|
||||
return runs, note
|
||||
}
|
||||
|
||||
// modelOfReplica is the model a replica id names: the control plane calls a
|
||||
// replica the model's name and a suffix.
|
||||
func modelOfReplica(id string) string {
|
||||
if i := strings.LastIndexByte(id, '-'); i > 0 {
|
||||
return id[:i]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func pidAlive(pid int) bool {
|
||||
err := syscall.Kill(pid, 0)
|
||||
return err == nil || errors.Is(err, syscall.EPERM)
|
||||
}
|
||||
|
||||
// --- the disks ---
|
||||
|
||||
func (p nodeProbe) readDisks() []diskUse {
|
||||
disks := []diskUse{{what: "weights", path: p.weightsDir}, {what: "engines", path: p.enginesDir}}
|
||||
for i := range disks {
|
||||
// In its own goroutine: statfs on a network mount that has gone away
|
||||
// does not return.
|
||||
type res struct {
|
||||
free, total uint64
|
||||
err error
|
||||
}
|
||||
ch := make(chan res, 1)
|
||||
go func(path string) {
|
||||
f, t, err := p.statfs(path)
|
||||
ch <- res{f, t, err}
|
||||
}(disks[i].path)
|
||||
select {
|
||||
case r := <-ch:
|
||||
disks[i].free, disks[i].total = r.free, r.total
|
||||
if r.err != nil {
|
||||
disks[i].err = r.err.Error()
|
||||
if errors.Is(r.err, os.ErrNotExist) {
|
||||
disks[i].err = "not there yet"
|
||||
}
|
||||
}
|
||||
case <-time.After(nodeCmdTimeout):
|
||||
disks[i].err = "does not answer"
|
||||
}
|
||||
}
|
||||
return disks
|
||||
}
|
||||
|
||||
func statfs(path string) (free, total uint64, err error) {
|
||||
var st syscall.Statfs_t
|
||||
if err := syscall.Statfs(path, &st); err != nil {
|
||||
return 0, 0, &os.PathError{Op: "statfs", Path: path, Err: err}
|
||||
}
|
||||
//nolint:unconvert // the field types differ between systems
|
||||
return uint64(st.Bavail) * uint64(st.Bsize), uint64(st.Blocks) * uint64(st.Bsize), nil
|
||||
}
|
||||
|
||||
// --- the problems ---
|
||||
|
||||
func (p nodeProbe) readProblems(ctx context.Context) ([]string, string) {
|
||||
out, err := p.run(ctx, "journalctl", "-u", "llmgw-agent", "-p", "warning", "--since", "-1h",
|
||||
"--no-pager", "-o", "cat", "-n", strconv.Itoa(nodeProblemLines))
|
||||
if err != nil && len(out) == 0 {
|
||||
return nil, "The journal cannot be read: " + err.Error()
|
||||
}
|
||||
lines := parseJournal(string(out), nodeProblemLines)
|
||||
if len(lines) == 0 && !p.root {
|
||||
return nil, "None that this user can see: run with sudo to read the agent's journal."
|
||||
}
|
||||
return lines, ""
|
||||
}
|
||||
|
||||
// parseJournal keeps the last n lines journalctl printed, without its own
|
||||
// remarks.
|
||||
func parseJournal(out string, n int) []string {
|
||||
var lines []string
|
||||
sc := bufio.NewScanner(strings.NewReader(out))
|
||||
sc.Buffer(make([]byte, 64*1024), 1024*1024)
|
||||
for sc.Scan() {
|
||||
l := strings.TrimSpace(sc.Text())
|
||||
if l == "" || strings.HasPrefix(l, "-- ") || strings.HasPrefix(l, "Hint: ") {
|
||||
continue
|
||||
}
|
||||
lines = append(lines, l)
|
||||
}
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return lines
|
||||
}
|
||||
|
||||
// runLocal runs a tool on this machine, bounded by nodeCmdTimeout. A tool
|
||||
// that is not installed is said so, not waited for.
|
||||
func runLocal(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
if _, err := exec.LookPath(name); err != nil {
|
||||
return nil, fmt.Errorf("%s is not installed", name)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
|
||||
defer cancel()
|
||||
//nolint:gosec // fixed tools and arguments, from this file
|
||||
return exec.CommandContext(ctx, name, args...).Output()
|
||||
}
|
||||
|
||||
// --- the panel ---
|
||||
|
||||
func (m consoleModel) nodeOverviewPanel(pw int) (string, string, string) {
|
||||
keys := "↑↓ choose · PgUp/PgDn scroll · Q leave · refreshed every three seconds"
|
||||
if m.nodeOv.at.IsZero() {
|
||||
return "Overview", "\n " + m.spin.View() + " reading this machine…", keys
|
||||
}
|
||||
return "Overview", renderNodeOverview(m.nodeOv, pw, time.Now()), keys
|
||||
}
|
||||
|
||||
// renderNodeOverview draws the overview from what was read.
|
||||
func renderNodeOverview(ov nodeOverview, pw int, now time.Time) string {
|
||||
var b strings.Builder
|
||||
head := func(s string) { b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render(s) + "\n") }
|
||||
kv := func(k, v string) { fmt.Fprintf(&b, " %-14s %s\n", sDim.Render(k), v) }
|
||||
line := func(s string) { b.WriteString(" " + s + "\n") }
|
||||
|
||||
head("Agent")
|
||||
a := ov.agent
|
||||
switch {
|
||||
case a.note != "":
|
||||
kv("llmgw-agent", sWarn.Render(a.note))
|
||||
default:
|
||||
state := a.state
|
||||
if a.sub != "" {
|
||||
state += " (" + a.sub + ")"
|
||||
}
|
||||
if a.state == "active" {
|
||||
state = sOK.Render(state)
|
||||
} else {
|
||||
state = sBad.Render(state)
|
||||
}
|
||||
if !a.since.IsZero() {
|
||||
state += sDim.Render(" since " + a.since.Format("2006-01-02 15:04") + ", " + uptime(now.Sub(a.since)) + " ago")
|
||||
}
|
||||
kv("llmgw-agent", state)
|
||||
}
|
||||
kv("llmgw", ov.version)
|
||||
|
||||
head("Control planes")
|
||||
if len(ov.planes) == 0 {
|
||||
line(sWarn.Render("None configured: agent.yaml names no server."))
|
||||
}
|
||||
for _, c := range ov.planes {
|
||||
state := sOK.Render("reachable")
|
||||
if c.err != "" {
|
||||
state = sBad.Render("✗ ") + truncate(c.err, max(20, pw-30))
|
||||
}
|
||||
if c.learned {
|
||||
state += sDim.Render(" (named by the leader)")
|
||||
}
|
||||
fmt.Fprintf(&b, " %-22s %s\n", c.addr, state)
|
||||
}
|
||||
|
||||
head("GPUs")
|
||||
if len(ov.gpus) == 0 {
|
||||
line(sDim.Render(ov.gpuNote))
|
||||
} else {
|
||||
var rows [][]string
|
||||
for _, g := range ov.gpus {
|
||||
busy, mem, temp := "—", "—", "—"
|
||||
if g.util >= 0 {
|
||||
busy = fmt.Sprintf("%.0f%%", g.util)
|
||||
}
|
||||
if g.memTotalMB > 0 {
|
||||
mem = fmt.Sprintf("%.1f/%.0f GB", g.memUsedMB/1024, g.memTotalMB/1024)
|
||||
}
|
||||
if g.tempC > 0 {
|
||||
temp = fmt.Sprintf("%.0f°C", g.tempC)
|
||||
}
|
||||
models := strings.Join(g.models, ", ")
|
||||
if models == "" {
|
||||
models = sDim.Render("free")
|
||||
}
|
||||
rows = append(rows, []string{strconv.Itoa(g.index), g.name, busy, mem, temp, models})
|
||||
}
|
||||
b.WriteString(indent(table([]string{"GPU", "CARD", "BUSY", "MEMORY", "TEMP", "MODELS"}, rows, -1, pw-2, false), " ") + "\n")
|
||||
}
|
||||
|
||||
head("Engines running here")
|
||||
switch {
|
||||
case len(ov.engines) > 0:
|
||||
var rows [][]string
|
||||
for _, r := range ov.engines {
|
||||
// The process by its pid, marked when it is no longer there.
|
||||
proc := sOK.Render(strconv.Itoa(r.pid))
|
||||
if !r.alive {
|
||||
proc = sBad.Render(strconv.Itoa(r.pid) + " gone")
|
||||
}
|
||||
gpus := strings.Join(discovery.DeviceLabels(r.gpus), ",")
|
||||
if gpus == "" {
|
||||
gpus = sDim.Render("CPU")
|
||||
}
|
||||
up := "—"
|
||||
if !r.started.IsZero() {
|
||||
up = uptime(now.Sub(r.started))
|
||||
}
|
||||
port := "—"
|
||||
if r.port > 0 {
|
||||
port = strconv.Itoa(r.port)
|
||||
}
|
||||
rows = append(rows, []string{r.replica, r.model, r.engine, gpus, port, proc, up})
|
||||
}
|
||||
b.WriteString(indent(table([]string{"REPLICA", "MODEL", "ENGINE", "GPUS", "PORT", "PID", "UP"}, rows, -1, pw-2, false), " ") + "\n")
|
||||
if ov.enginesNote != "" {
|
||||
line(sWarn.Render(ov.enginesNote))
|
||||
}
|
||||
case ov.enginesNote != "":
|
||||
line(sWarn.Render(ov.enginesNote))
|
||||
default:
|
||||
line(sDim.Render("None: the control plane has placed nothing here."))
|
||||
}
|
||||
|
||||
head("Disk")
|
||||
for _, d := range ov.disks {
|
||||
use := sWarn.Render(d.err)
|
||||
if d.err == "" {
|
||||
use = fmt.Sprintf("%s free of %s", diskSize(d.free), diskSize(d.total))
|
||||
if d.total > 0 && d.free*10 < d.total {
|
||||
use = sWarn.Render(use)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, " %-14s %-28s %s\n", sDim.Render(d.what), d.path, use)
|
||||
}
|
||||
|
||||
head("Recent problems")
|
||||
switch {
|
||||
case ov.problemsNote != "":
|
||||
line(sDim.Render(ov.problemsNote))
|
||||
case len(ov.problems) == 0:
|
||||
line(sOK.Render("None in the last hour."))
|
||||
default:
|
||||
for _, l := range ov.problems {
|
||||
l = truncateWidth(l, max(20, pw-4))
|
||||
if strings.Contains(l, "level=ERROR") {
|
||||
l = sBad.Render(l)
|
||||
} else {
|
||||
l = sWarn.Render(l)
|
||||
}
|
||||
line(l)
|
||||
}
|
||||
}
|
||||
b.WriteString("\n " + sDim.Render("read at "+ov.at.Format("15:04:05")) + "\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// uptime is a duration as people say it: 3d4h, 2h15m, 40s.
|
||||
func uptime(d time.Duration) string {
|
||||
d = d.Round(time.Second)
|
||||
switch {
|
||||
case d < 0:
|
||||
return "0s"
|
||||
case d >= 24*time.Hour:
|
||||
return fmt.Sprintf("%dd%dh", d/(24*time.Hour), (d%(24*time.Hour))/time.Hour)
|
||||
case d >= time.Hour:
|
||||
return fmt.Sprintf("%dh%dm", d/time.Hour, (d%time.Hour)/time.Minute)
|
||||
case d >= time.Minute:
|
||||
return fmt.Sprintf("%dm%ds", d/time.Minute, (d%time.Minute)/time.Second)
|
||||
}
|
||||
return d.String()
|
||||
}
|
||||
|
||||
// diskSize is a byte count in the unit disks are sold in.
|
||||
func diskSize(n uint64) string {
|
||||
switch {
|
||||
case n >= 1<<40:
|
||||
return fmt.Sprintf("%.1f TB", float64(n)/(1<<40))
|
||||
case n >= 1<<30:
|
||||
return fmt.Sprintf("%.0f GB", float64(n)/(1<<30))
|
||||
}
|
||||
return fmt.Sprintf("%.0f MB", float64(n)/(1<<20))
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/charmbracelet/x/ansi"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
func writeNodeFile(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewDialsTheConfiguredControlPlanesThenTheLeadersOwn(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, "control-planes.json")
|
||||
writeNodeFile(t, file, `["https://10.0.0.150:7443","10.0.0.140:7443","https://cp3.example/agent"]`)
|
||||
got := readControlPlanes(file, []string{"10.0.0.140:7443", "http://10.0.0.141"})
|
||||
want := []planeCheck{{addr: "10.0.0.140:7443"}, {addr: "10.0.0.141:80"},
|
||||
{addr: "10.0.0.150:7443", learned: true}, {addr: "cp3.example:443", learned: true}}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %+v, want %+v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Errorf("%d: got %+v, want %+v", i, got[i], want[i])
|
||||
}
|
||||
}
|
||||
// Without the file, what agent.yaml says.
|
||||
if got := readControlPlanes(filepath.Join(dir, "none.json"), []string{"cp:7443"}); len(got) != 1 || got[0].addr != "cp:7443" {
|
||||
t.Errorf("without the file: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewReadsTheEnginesFromTheSupervisorsStateFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"shard_id":"qwen3-8b-ab12-0","replica_id":"qwen3-8b-ab12",
|
||||
"pid":4242,"started_at":"2026-10-19T08:00:00Z","spec":{"ReplicaID":"qwen3-8b-ab12","GPUIndices":[0,1],"HTTPPort":8101,
|
||||
"Profile":{"Name":"vllm-tp2","Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-bge-m3-7-0.json"), `{"replica_id":"bge-m3-7","pid":99,
|
||||
"spec":{"GPUIndices":[1002],"HTTPPort":8102,"Profile":{"Engine":"llamacpp"}}}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-x.json.tmp"), `{"replica_id":"half-written"}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "control-planes.json"), `[]`)
|
||||
runs, note := readShardStates(dir, func(pid int) bool { return pid == 4242 })
|
||||
if note != "" {
|
||||
t.Fatalf("note %q", note)
|
||||
}
|
||||
if len(runs) != 2 {
|
||||
t.Fatalf("got %+v", runs)
|
||||
}
|
||||
b, q := runs[0], runs[1]
|
||||
if q.replica != "qwen3-8b-ab12" || q.model != "qwen3-8b" || q.engine != "vllm" || q.port != 8101 || !q.alive ||
|
||||
len(q.gpus) != 2 || !q.started.Equal(time.Date(2026, 10, 19, 8, 0, 0, 0, time.UTC)) {
|
||||
t.Errorf("qwen: %+v", q)
|
||||
}
|
||||
// No served name: the model from the replica's id.
|
||||
if b.model != "bge-m3" || b.alive || b.engine != "llamacpp" {
|
||||
t.Errorf("bge: %+v", b)
|
||||
}
|
||||
if runs, note := readShardStates(filepath.Join(dir, "absent"), pidAlive); runs != nil || note != "" {
|
||||
t.Errorf("no work dir: %+v %q", runs, note)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewNamesTheModelOnEachGPU(t *testing.T) {
|
||||
gpus := []nodeGPU{{index: 0}, {index: 1}, {index: 2}}
|
||||
runs := []engineRun{
|
||||
{model: "qwen3-8b", gpus: []int{0, 1}},
|
||||
{model: "bge-m3", gpus: []int{discovery.MIGIndex(1, 2)}},
|
||||
{model: "qwen3-8b", gpus: []int{1}},
|
||||
}
|
||||
got := modelsOnGPUs(gpus, runs)
|
||||
if strings.Join(got[0].models, ",") != "qwen3-8b" || strings.Join(got[1].models, ",") != "qwen3-8b,bge-m3" ||
|
||||
len(got[2].models) != 0 {
|
||||
t.Errorf("got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewReadsWhatSystemdSaysOfTheAgent(t *testing.T) {
|
||||
s := parseSystemctlShow("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:12:03 UTC\n")
|
||||
if s.state != "active" || s.sub != "running" || s.note != "" || s.since.IsZero() || s.since.Minute() != 12 {
|
||||
t.Errorf("got %+v", s)
|
||||
}
|
||||
if s := parseSystemctlShow("LoadState=not-found\nActiveState=inactive\nActiveEnterTimestamp=\n"); s.note == "" {
|
||||
t.Errorf("a missing service: %+v", s)
|
||||
}
|
||||
if s := parseSystemctlShow("LoadState=loaded\nActiveState=failed\nSubState=failed\nActiveEnterTimestamp=n/a\n"); s.state != "failed" ||
|
||||
!s.since.IsZero() {
|
||||
t.Errorf("a failed service: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewKeepsTheLastWarningsOfTheJournal(t *testing.T) {
|
||||
var b strings.Builder
|
||||
for i := 1; i <= 9; i++ {
|
||||
b.WriteString("level=WARN msg=w" + string(rune('0'+i)) + "\n")
|
||||
}
|
||||
got := parseJournal(b.String(), 6)
|
||||
if len(got) != 6 || got[0] != "level=WARN msg=w4" || got[5] != "level=WARN msg=w9" {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
if got := parseJournal("-- No entries --\n", 6); len(got) != 0 {
|
||||
t.Errorf("no entries: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// fakeProbe is a node with two cards, one engine and a broken second control
|
||||
// plane, and no tool of the machine running the test touched.
|
||||
func fakeProbe(t *testing.T) nodeProbe {
|
||||
dir := t.TempDir()
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"replica_id":"qwen3-8b-ab12","pid":4242,
|
||||
"started_at":"2026-10-19T08:00:00Z","spec":{"GPUIndices":[0],"HTTPPort":8101,"Profile":{"Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
|
||||
return nodeProbe{
|
||||
configured: []string{"10.0.0.140:7443", "10.0.0.150:7443"},
|
||||
planesFile: filepath.Join(dir, "control-planes.json"),
|
||||
workDir: dir, weightsDir: "/var/lib/llmgw-weights", enginesDir: "/opt/llmgw/engines", root: true,
|
||||
run: func(_ context.Context, name string, _ ...string) ([]byte, error) {
|
||||
switch name {
|
||||
case "systemctl":
|
||||
return []byte("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:00:00 UTC\n"), nil
|
||||
case "journalctl":
|
||||
return []byte("level=WARN msg=\"the control plane does not answer\" server=10.0.0.150:7443\n"), nil
|
||||
}
|
||||
return nil, errors.New(name + " is not installed")
|
||||
},
|
||||
dial: func(_ context.Context, addr string) error {
|
||||
if addr == "10.0.0.150:7443" {
|
||||
return errors.New("dial tcp 10.0.0.150:7443: connect: connection refused")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
gpus: func(context.Context) []discovery.GPUSensor {
|
||||
return []discovery.GPUSensor{
|
||||
{Index: 0, TempC: 61, UtilPercent: 87, MemoryUsedMB: 20480, MemoryTotalMB: 24576},
|
||||
{Index: 1, TempC: 35, UtilPercent: 0, MemoryUsedMB: 300, MemoryTotalMB: 24576},
|
||||
}
|
||||
},
|
||||
cards: func(context.Context) ([]discovery.GPU, error) {
|
||||
return []discovery.GPU{{Index: 0, Name: "NVIDIA RTX A5000"}, {Index: 1, Name: "NVIDIA RTX A5000"}}, nil
|
||||
},
|
||||
alive: func(int) bool { return true },
|
||||
statfs: func(string) (uint64, uint64, error) { return 120 << 30, 500 << 30, nil },
|
||||
now: func() time.Time { return time.Date(2026, 10, 19, 11, 12, 0, 0, time.UTC) },
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewShowsWhatRunsOnTheMachine(t *testing.T) {
|
||||
ov := fakeProbe(t).gather(context.Background())
|
||||
out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
|
||||
t.Log("\n" + out)
|
||||
for _, want := range []string{
|
||||
"llmgw-agent", "active (running)", "3h12m ago",
|
||||
"10.0.0.140:7443", "reachable", "connection refused",
|
||||
"NVIDIA RTX A5000", "87%", "20.0/24 GB", "61°C", "qwen3-8b", "free",
|
||||
"qwen3-8b-ab12", "vllm", "8101", "4242", "3h12m",
|
||||
"/var/lib/llmgw-weights", "120 GB free of 500 GB", "/opt/llmgw/engines",
|
||||
"the control plane does not answer",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("the overview lacks %q", want)
|
||||
}
|
||||
}
|
||||
if ov.names[0] != "NVIDIA RTX A5000" {
|
||||
t.Errorf("the names are not kept for the next reading: %v", ov.names)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeWithoutAGPUSaysSoInOneLine(t *testing.T) {
|
||||
p := fakeProbe(t)
|
||||
p.gpus = func(context.Context) []discovery.GPUSensor { return nil }
|
||||
p.cards = func(context.Context) ([]discovery.GPU, error) { return nil, nil }
|
||||
p.run = func(context.Context, string, ...string) ([]byte, error) { return nil, errors.New("not installed") }
|
||||
ov := p.gather(context.Background())
|
||||
out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
|
||||
if !strings.Contains(out, "No GPU on this machine.") {
|
||||
t.Errorf("no line saying there is no GPU:\n%s", out)
|
||||
}
|
||||
if strings.Contains(out, "BUSY") {
|
||||
t.Errorf("an empty GPU table:\n%s", out)
|
||||
}
|
||||
// No systemctl or journalctl is said, not a crash.
|
||||
if !strings.Contains(out, "systemctl is not available") || !strings.Contains(out, "The journal cannot be read") {
|
||||
t.Errorf("missing tools are not said:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesConsoleOpensOnItsOverview(t *testing.T) {
|
||||
m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "agent.yaml"))
|
||||
if m.menu != cOverview || !m.entryOK(cOverview) {
|
||||
t.Fatalf("menu %d", m.menu)
|
||||
}
|
||||
_, body, _ := m.panel()
|
||||
if !strings.Contains(body, "reading this machine") {
|
||||
t.Errorf("before the first reading: %q", body)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,664 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"go.yaml.in/yaml/v3"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// How people sign in, as the console shows and sets it: single sign-on
|
||||
// through Entra ID or another OpenID provider, an Active Directory checked
|
||||
// over LDAPS, and provisioning by SCIM. The secrets these need are kept in
|
||||
// the store, so every control plane has them without a file put on each,
|
||||
// and a directory is tried before its settings are kept.
|
||||
|
||||
// inChainRule is AD's matching rule for nested group membership.
|
||||
const inChainRule = "1.2.840.113556.1.4.1941"
|
||||
|
||||
type signInState struct {
|
||||
view adminapi.AuthView
|
||||
err error
|
||||
loaded bool
|
||||
// ldap and oidc are the blocks the cluster's settings hold now, for the
|
||||
// forms to start from.
|
||||
ldap config.LDAP
|
||||
oidc config.OIDC
|
||||
}
|
||||
|
||||
type consoleSignInMsg struct {
|
||||
view adminapi.AuthView
|
||||
ldap config.LDAP
|
||||
oidc config.OIDC
|
||||
err error
|
||||
}
|
||||
|
||||
// signInRows are the panel's rows, one per way in.
|
||||
var signInRows = []string{"oidc", "ldap", "scim"}
|
||||
|
||||
func (m consoleModel) fetchSignIn() tea.Cmd {
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
var msg consoleSignInMsg
|
||||
if msg.err = c.get(ctx, "/api/v1/auth", &msg.view); msg.err != nil {
|
||||
return msg
|
||||
}
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err == nil {
|
||||
msg.ldap, msg.oidc = authBlocks(cur.Body)
|
||||
}
|
||||
return msg
|
||||
}
|
||||
}
|
||||
|
||||
// authBlocks are auth.ldap and auth.oidc as a settings document has them.
|
||||
func authBlocks(body string) (config.LDAP, config.OIDC) {
|
||||
var doc struct {
|
||||
Auth struct {
|
||||
LDAP yaml.Node `yaml:"ldap"`
|
||||
OIDC yaml.Node `yaml:"oidc"`
|
||||
} `yaml:"auth"`
|
||||
}
|
||||
var l config.LDAP
|
||||
var o config.OIDC
|
||||
if yaml.Unmarshal([]byte(body), &doc) != nil {
|
||||
return l, o
|
||||
}
|
||||
if doc.Auth.LDAP.Kind != 0 {
|
||||
if b, err := yaml.Marshal(&doc.Auth.LDAP); err == nil {
|
||||
// Read as kept, whether or not it would validate now.
|
||||
l, _ = config.ParseLDAP(b)
|
||||
}
|
||||
}
|
||||
if doc.Auth.OIDC.Kind != 0 {
|
||||
if m, err := yaml.Marshal(&doc.Auth.OIDC); err == nil {
|
||||
if cfg, err := config.FromMap(map[string]any{"auth": map[string]any{"oidc": yamlMap(m)}}); err == nil {
|
||||
o = cfg.Auth.OIDC
|
||||
}
|
||||
}
|
||||
}
|
||||
return l, o
|
||||
}
|
||||
|
||||
func yamlMap(b []byte) map[string]any {
|
||||
var m map[string]any
|
||||
_ = yaml.Unmarshal(b, &m)
|
||||
return m
|
||||
}
|
||||
|
||||
func (m consoleModel) signInPanel(pw int) (string, string, string) {
|
||||
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||
if m.inPanel {
|
||||
keys = "↑↓ row · Enter actions · E Entra ID · D Active Directory · T try the directory · ← back · Q leave"
|
||||
}
|
||||
s := m.signIn
|
||||
var b strings.Builder
|
||||
b.WriteString("\n " + sDim.Render("How people sign in besides a password kept here. E for Entra ID, D for Active Directory.") + "\n\n")
|
||||
if s.err != nil {
|
||||
b.WriteString(" " + sBad.Render(s.err.Error()))
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
if !s.loaded {
|
||||
b.WriteString(" " + sDim.Render("asking the control plane…"))
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
v := s.view
|
||||
local := "passwords kept here; accounts made by an administrator"
|
||||
if v.SignupOpen {
|
||||
local = "passwords kept here; anyone may sign up, then wait for approval"
|
||||
}
|
||||
b.WriteString(" " + sTitle.Render("LOCAL") + " " + sDim.Render(local) + "\n\n")
|
||||
|
||||
var rows [][]string
|
||||
for _, r := range signInRows {
|
||||
switch r {
|
||||
case "oidc":
|
||||
if v.OIDC == nil {
|
||||
rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", "off", "E to set up"})
|
||||
continue
|
||||
}
|
||||
state := "on"
|
||||
if v.OIDC.NoPasswords {
|
||||
state = "on, the only way in"
|
||||
}
|
||||
rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", state, or(v.OIDC.Label, v.OIDC.Issuer)})
|
||||
case "ldap":
|
||||
if v.LDAP == nil {
|
||||
rows = append(rows, []string{"Active Directory (LDAP)", "off", "D to set up"})
|
||||
continue
|
||||
}
|
||||
state := "on"
|
||||
if v.LDAP.Off != "" {
|
||||
state = "broken here"
|
||||
}
|
||||
rows = append(rows, []string{"Active Directory (LDAP)", state, strings.Join(v.LDAP.URLs, " ")})
|
||||
case "scim":
|
||||
state, detail := "off", "Enter to make a token"
|
||||
if v.SCIM {
|
||||
state, detail = "on", "at /scim/v2"
|
||||
}
|
||||
rows = append(rows, []string{"Provisioning (SCIM)", state, detail})
|
||||
}
|
||||
}
|
||||
b.WriteString(table([]string{"WAY IN", "STATE", ""}, rows, m.row, pw, m.inPanel) + "\n")
|
||||
|
||||
if v.OIDC != nil {
|
||||
b.WriteString("\n " + sTitle.Render("SINGLE SIGN-ON") + "\n")
|
||||
b.WriteString(" issuer " + v.OIDC.Issuer + "\n")
|
||||
b.WriteString(" redirect " + redirectURL(v.BaseURL) + "\n")
|
||||
b.WriteString(" " + sDim.Render(pickText(v.OIDC.CreateAccounts, "accounts made at first sign-in", "only people with an account sign in")) + "\n")
|
||||
}
|
||||
if l := v.LDAP; l != nil {
|
||||
b.WriteString("\n " + sTitle.Render("ACTIVE DIRECTORY") + " " + sDim.Render(l.BaseDN) + "\n")
|
||||
if l.Off != "" {
|
||||
b.WriteString(" " + sBad.Render("off on this control plane: "+l.Off) + "\n")
|
||||
}
|
||||
switch ls := l.LastSync; {
|
||||
case ls == nil:
|
||||
b.WriteString(" " + sDim.Render("not read yet: the leader reads it every sync_interval") + "\n")
|
||||
case ls.Error != "":
|
||||
b.WriteString(" " + sBad.Render("last read "+agoText(ls.At)+": "+ls.Error) + "\n")
|
||||
default:
|
||||
b.WriteString(fmt.Sprintf(" last read %s: %d people there, %d accounts kept in step, %d suspended, %d reinstated\n",
|
||||
agoText(ls.At), ls.People, ls.Accounts, ls.Suspended, ls.Reinstated))
|
||||
}
|
||||
}
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
|
||||
func agoText(t time.Time) string { return time.Since(t).Round(time.Second).String() + " ago" }
|
||||
|
||||
func pickText(b bool, yes, no string) string {
|
||||
if b {
|
||||
return yes
|
||||
}
|
||||
return no
|
||||
}
|
||||
|
||||
// redirectURL is what an identity provider sends people back to.
|
||||
func redirectURL(base string) string {
|
||||
if base == "" {
|
||||
return "https://<this gateway>/login/oidc/callback"
|
||||
}
|
||||
return strings.TrimRight(base, "/") + "/login/oidc/callback"
|
||||
}
|
||||
|
||||
// signInMenu is what can be done to one way in.
|
||||
func (m consoleModel) signInMenu(row string) *menuDialog {
|
||||
v := m.signIn.view
|
||||
switch row {
|
||||
case "oidc":
|
||||
d := &menuDialog{title: "Single sign-on", items: []menuItem{
|
||||
{"Set up Entra ID…", func(m *consoleModel) tea.Cmd { m.entraForm(); return nil }},
|
||||
}}
|
||||
if v.OIDC != nil {
|
||||
d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("oidc", "Single sign-on", "People sign in with a password kept here again; accounts made through the provider have none.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
case "ldap":
|
||||
d := &menuDialog{title: "Active Directory", items: []menuItem{
|
||||
{"Set up…", func(m *consoleModel) tea.Cmd { m.directoryForm(); return nil }},
|
||||
}}
|
||||
if v.LDAP != nil {
|
||||
d.items = append(d.items,
|
||||
menuItem{"Try it", func(m *consoleModel) tea.Cmd { return m.tryDirectory() }},
|
||||
menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("ldap", "Active Directory", "Its people can no longer sign in; their accounts stay, and are no longer kept in step.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
default:
|
||||
d := &menuDialog{title: "Provisioning (SCIM)", items: []menuItem{
|
||||
{"Make a new token", func(m *consoleModel) tea.Cmd { return m.scimToken() }},
|
||||
}}
|
||||
if v.SCIM {
|
||||
d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("scim", "Provisioning", "The provider can no longer make, change or suspend accounts here.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
}
|
||||
}
|
||||
|
||||
// signInRowKey is the way in the cursor is on.
|
||||
func (m consoleModel) signInRowKey() string {
|
||||
if m.row >= 0 && m.row < len(signInRows) {
|
||||
return signInRows[m.row]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// directoryForm sets up auth.ldap, starting from what is kept.
|
||||
func (m *consoleModel) directoryForm() {
|
||||
l := m.signIn.ldap
|
||||
allowed := ""
|
||||
if f := l.UserFilter; strings.HasPrefix(f, "(memberOf:"+inChainRule+":=") {
|
||||
allowed = strings.TrimSuffix(strings.TrimPrefix(f, "(memberOf:"+inChainRule+":="), ")")
|
||||
} else {
|
||||
allowed = f
|
||||
}
|
||||
pwHint := "the service account's password"
|
||||
if l.BindPasswordFile != "" {
|
||||
pwHint = "empty keeps the one kept"
|
||||
}
|
||||
caHint := "a PEM file on this machine; empty uses the system's"
|
||||
if l.CA != "" || l.CAFile != "" {
|
||||
caHint = "empty keeps the one kept; - for the system's"
|
||||
}
|
||||
create := "yes"
|
||||
if l.Enabled() && !l.CreateAccounts {
|
||||
create = "no"
|
||||
}
|
||||
var teams []string
|
||||
for g, place := range l.Memberships {
|
||||
teams = append(teams, g+"="+place)
|
||||
}
|
||||
pw := textF("password", "", pwHint)
|
||||
pw.text.Mask = true
|
||||
m.openForm(&formDialog{title: "Active Directory",
|
||||
note: "People sign in with their AD name and password, checked over LDAPS. Tried before it is kept.",
|
||||
fields: []formField{
|
||||
textF("servers", strings.Join(l.URLs, " "), "ldaps://dc1.corp.example ldaps://dc2.corp.example"),
|
||||
textF("people under", l.BaseDN, "OU=Staff,DC=corp,DC=example"),
|
||||
textF("service account", l.BindDN, "CN=svc-llmgw,OU=Service,DC=corp,DC=example"),
|
||||
pw,
|
||||
textF("CA certificate", "", caHint),
|
||||
textF("who may sign in", allowed, "a group's DN, nested groups count; empty is everyone under it"),
|
||||
textF("administrators", strings.Join(l.Roles["admin"], ", "), "group names or DNs"),
|
||||
textF("operators", strings.Join(l.Roles["operator"], ", "), "group names or DNs"),
|
||||
textF("teams", strings.Join(teams, "; "), "ml-team=team:Acme/ml; sales=org:Sales"),
|
||||
textF("label", l.Label, "the form says: Sign in with your LABEL account"),
|
||||
choiceF("make accounts", []string{"yes", "no"}, create),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
next, err := directoryFromForm(l, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m.saveDirectory(next, v[3]), nil
|
||||
}})
|
||||
}
|
||||
|
||||
// directoryFromForm is the auth.ldap block the form's values make, over
|
||||
// what was kept.
|
||||
func directoryFromForm(kept config.LDAP, v []string) (config.LDAP, error) {
|
||||
l := kept
|
||||
l.URLs = strings.Fields(strings.ReplaceAll(v[0], ",", " "))
|
||||
l.BaseDN, l.BindDN = v[1], v[2]
|
||||
if v[3] != "" || l.BindPasswordFile == "" {
|
||||
l.BindPasswordFile = adminapi.AuthSecretRef("ldap-bind")
|
||||
}
|
||||
switch ca := v[4]; ca {
|
||||
case "":
|
||||
case "-":
|
||||
l.CA, l.CAFile = "", ""
|
||||
default:
|
||||
b, err := os.ReadFile(ca) //nolint:gosec // a file the operator names
|
||||
if err != nil {
|
||||
return l, fmt.Errorf("CA certificate: %w", err)
|
||||
}
|
||||
l.CA, l.CAFile = string(b), ""
|
||||
}
|
||||
switch g := v[5]; {
|
||||
case g == "":
|
||||
l.UserFilter = ""
|
||||
case strings.HasPrefix(g, "("):
|
||||
l.UserFilter = g
|
||||
case strings.Contains(g, "="):
|
||||
l.UserFilter = "(memberOf:" + inChainRule + ":=" + g + ")"
|
||||
default:
|
||||
return l, errors.New("who may sign in: a group's DN (CN=…,OU=…,DC=…), or an LDAP filter in parentheses")
|
||||
}
|
||||
l.Roles = map[string][]string{}
|
||||
for role, i := range map[string]int{"admin": 6, "operator": 7} {
|
||||
for _, g := range strings.Split(v[i], ",") {
|
||||
if g = strings.TrimSpace(g); g != "" {
|
||||
l.Roles[role] = append(l.Roles[role], g)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(l.Roles) == 0 {
|
||||
l.Roles = nil
|
||||
}
|
||||
l.Memberships = nil
|
||||
for _, pair := range strings.Split(v[8], ";") {
|
||||
if pair = strings.TrimSpace(pair); pair == "" {
|
||||
continue
|
||||
}
|
||||
g, place, ok := strings.Cut(pair, "=")
|
||||
if !ok {
|
||||
return l, fmt.Errorf("teams: %q is not GROUP=team:ORG/TEAM", pair)
|
||||
}
|
||||
if l.Memberships == nil {
|
||||
l.Memberships = map[string]string{}
|
||||
}
|
||||
l.Memberships[strings.TrimSpace(g)] = strings.TrimSpace(place)
|
||||
}
|
||||
l.Label, l.CreateAccounts = v[9], v[10] == "yes"
|
||||
return l, l.Validate()
|
||||
}
|
||||
|
||||
// ldapYAML is a block as the cluster's settings write it.
|
||||
func ldapYAML(l config.LDAP) string {
|
||||
m := map[string]any{"urls": l.URLs, "bind_dn": l.BindDN, "bind_password_file": l.BindPasswordFile,
|
||||
"base_dn": l.BaseDN, "create_accounts": l.CreateAccounts}
|
||||
put := func(k, v string) {
|
||||
if v != "" {
|
||||
m[k] = v
|
||||
}
|
||||
}
|
||||
put("ca", l.CA)
|
||||
put("ca_file", l.CAFile)
|
||||
put("user_filter", l.UserFilter)
|
||||
put("group_base_dn", l.GroupBaseDN)
|
||||
put("label", l.Label)
|
||||
if l.StartTLS {
|
||||
m["start_tls"] = true
|
||||
}
|
||||
if len(l.Roles) > 0 {
|
||||
m["roles"] = l.Roles
|
||||
}
|
||||
if len(l.Memberships) > 0 {
|
||||
m["memberships"] = l.Memberships
|
||||
}
|
||||
if l.SyncInterval != 0 {
|
||||
m["sync_interval"] = l.SyncInterval.String()
|
||||
}
|
||||
if l.Timeout != 0 {
|
||||
m["timeout"] = l.Timeout.String()
|
||||
}
|
||||
b, _ := yaml.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// saveDirectory tries the block, and only if every step passes keeps the
|
||||
// password and puts the block in the cluster's settings.
|
||||
func (m *consoleModel) saveDirectory(l config.LDAP, password string) tea.Cmd {
|
||||
m.busy = "Active Directory"
|
||||
c, block := m.client, ldapYAML(l)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
var tv adminapi.LDAPTestView
|
||||
if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test",
|
||||
adminapi.LDAPTestRequest{Settings: block, Password: password}, &tv); err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
report := checksText(tv.Checks)
|
||||
if !tv.OK {
|
||||
return actionDoneMsg{title: "Active Directory", err: errors.New("not kept: the directory did not pass\n\n" + report)}
|
||||
}
|
||||
if password != "" {
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/ldap-bind", adminapi.SecretRequest{Value: password}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.ldap from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "ldap"}, block)
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
return actionDoneMsg{title: "Active Directory", text: report + "\n\n" + said}
|
||||
}
|
||||
}
|
||||
|
||||
// tryDirectory tries the block in force.
|
||||
func (m *consoleModel) tryDirectory() tea.Cmd {
|
||||
m.busy = "Trying the directory"
|
||||
c, block := m.client, ldapYAML(m.signIn.ldap)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
var tv adminapi.LDAPTestView
|
||||
if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test", adminapi.LDAPTestRequest{Settings: block}, &tv); err != nil {
|
||||
return actionDoneMsg{title: "The directory", err: err}
|
||||
}
|
||||
if !tv.OK {
|
||||
return actionDoneMsg{title: "The directory", err: errors.New("it did not pass\n\n" + checksText(tv.Checks))}
|
||||
}
|
||||
return actionDoneMsg{title: "The directory", text: checksText(tv.Checks)}
|
||||
}
|
||||
}
|
||||
|
||||
func checksText(checks []adminapi.CheckView) string {
|
||||
var b strings.Builder
|
||||
for i, c := range checks {
|
||||
if i > 0 {
|
||||
b.WriteString("\n")
|
||||
}
|
||||
mark := sOK.Render("✓")
|
||||
if !c.OK {
|
||||
mark = sBad.Render("✗")
|
||||
}
|
||||
fmt.Fprintf(&b, "%s %s: %s", mark, c.Step, c.Detail)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// entraForm sets up single sign-on through Entra ID, or another provider
|
||||
// when an issuer address is given in place of the tenant.
|
||||
func (m *consoleModel) entraForm() {
|
||||
o := m.signIn.oidc
|
||||
tenant := strings.TrimSuffix(strings.TrimPrefix(o.Issuer, "https://login.microsoftonline.com/"), "/v2.0")
|
||||
secretHint := "the app registration's client secret"
|
||||
if o.ClientSecretFile != "" {
|
||||
secretHint = "empty keeps the one kept"
|
||||
}
|
||||
create, only := "yes", "no"
|
||||
if o.Enabled() && !o.CreateAccounts {
|
||||
create = "no"
|
||||
}
|
||||
if o.NoPasswords {
|
||||
only = "yes"
|
||||
}
|
||||
secret := textF("client secret", "", secretHint)
|
||||
secret.text.Mask = true
|
||||
m.openForm(&formDialog{title: "Single sign-on: Entra ID",
|
||||
note: "Register an app in Entra with the redirect address " + redirectURL(m.signIn.view.BaseURL) +
|
||||
", and a groups claim of the groups assigned to it.",
|
||||
fields: []formField{
|
||||
textF("tenant", tenant, "the directory (tenant) ID; or another provider's issuer URL"),
|
||||
textF("client ID", o.ClientID, "the application (client) ID"),
|
||||
secret,
|
||||
textF("administrators", strings.Join(o.Roles["admin"], ", "), "group object IDs, as Entra sends them"),
|
||||
textF("operators", strings.Join(o.Roles["operator"], ", "), "group object IDs"),
|
||||
textF("label", or(o.Label, "Microsoft"), "the button says: Sign in with LABEL"),
|
||||
choiceF("make accounts", []string{"yes", "no"}, create),
|
||||
choiceF("only this way", []string{"no", "yes"}, only),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
next, err := oidcFromForm(o, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m.saveOIDC(next, v[2]), nil
|
||||
}})
|
||||
}
|
||||
|
||||
func oidcFromForm(kept config.OIDC, v []string) (config.OIDC, error) {
|
||||
o := kept
|
||||
switch t := strings.TrimSpace(v[0]); {
|
||||
case t == "":
|
||||
return o, errors.New("tenant: the directory (tenant) ID, or an issuer URL")
|
||||
case strings.Contains(t, "://"):
|
||||
o.Issuer = t
|
||||
default:
|
||||
o.Issuer = "https://login.microsoftonline.com/" + t + "/v2.0"
|
||||
}
|
||||
if o.ClientID = v[1]; o.ClientID == "" {
|
||||
return o, errors.New("client ID: the application (client) ID")
|
||||
}
|
||||
if v[2] != "" || o.ClientSecretFile == "" {
|
||||
if v[2] == "" {
|
||||
return o, errors.New("client secret: the app registration's secret")
|
||||
}
|
||||
o.ClientSecretFile = adminapi.AuthSecretRef("oidc-client-secret")
|
||||
}
|
||||
o.Roles = map[string][]string{}
|
||||
for role, i := range map[string]int{"admin": 3, "operator": 4} {
|
||||
for _, g := range strings.Split(v[i], ",") {
|
||||
if g = strings.TrimSpace(g); g != "" {
|
||||
o.Roles[role] = append(o.Roles[role], g)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(o.Roles) == 0 {
|
||||
o.Roles = nil
|
||||
}
|
||||
o.Label, o.CreateAccounts, o.NoPasswords = v[5], v[6] == "yes", v[7] == "yes"
|
||||
return o, o.Validate()
|
||||
}
|
||||
|
||||
func oidcYAML(o config.OIDC) string {
|
||||
m := map[string]any{"issuer": o.Issuer, "client_id": o.ClientID, "client_secret_file": o.ClientSecretFile,
|
||||
"create_accounts": o.CreateAccounts}
|
||||
if o.Label != "" {
|
||||
m["label"] = o.Label
|
||||
}
|
||||
if o.NoPasswords {
|
||||
m["no_passwords"] = true
|
||||
}
|
||||
if o.RedirectURL != "" {
|
||||
m["redirect_url"] = o.RedirectURL
|
||||
}
|
||||
if len(o.Scopes) > 0 {
|
||||
m["scopes"] = o.Scopes
|
||||
}
|
||||
if o.GroupsClaim != "" {
|
||||
m["groups_claim"] = o.GroupsClaim
|
||||
}
|
||||
if len(o.Roles) > 0 {
|
||||
m["roles"] = o.Roles
|
||||
}
|
||||
if len(o.Memberships) > 0 {
|
||||
m["memberships"] = o.Memberships
|
||||
}
|
||||
b, _ := yaml.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func (m *consoleModel) saveOIDC(o config.OIDC, secret string) tea.Cmd {
|
||||
m.busy = "Single sign-on"
|
||||
c, block, redirect := m.client, oidcYAML(o), redirectURL(m.signIn.view.BaseURL)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
if secret != "" {
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/oidc-client-secret", adminapi.SecretRequest{Value: secret}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Single sign-on", err: err}
|
||||
}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.oidc from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "oidc"}, block)
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Single sign-on", err: err}
|
||||
}
|
||||
return actionDoneMsg{title: "Single sign-on", text: said + "\n\nIn the app registration the redirect address is " + redirect +
|
||||
". For people in more than 200 groups, emit the groups assigned to the application."}
|
||||
}
|
||||
}
|
||||
|
||||
// scimToken makes a new provisioning token, keeps it, and shows it once.
|
||||
func (m *consoleModel) scimToken() tea.Cmd {
|
||||
m.busy = "Provisioning"
|
||||
c, base := m.client, m.signIn.view.BaseURL
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
token := hex.EncodeToString(b)
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/scim-token", adminapi.SecretRequest{Value: token}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.scim from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "scim", "token_file"}, adminapi.AuthSecretRef("scim-token"))
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
url := "https://<this gateway>/scim/v2"
|
||||
if base != "" {
|
||||
url = strings.TrimRight(base, "/") + "/scim/v2"
|
||||
}
|
||||
return actionDoneMsg{title: "Provisioning", text: "Give the provider the tenant URL " + url + " and this token, shown only now:\n\n " +
|
||||
token + "\n\n" + said}
|
||||
}
|
||||
}
|
||||
|
||||
// signInOffConfirm turns one way in off, after asking.
|
||||
func (m *consoleModel) signInOffConfirm(key, title, text string) {
|
||||
m.confirmD = &confirmDialog{title: "Turn off " + strings.ToLower(title), text: text,
|
||||
yes: func(m *consoleModel) tea.Cmd {
|
||||
m.busy = title
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
said, err := changeClusterSettings(ctx, c, "auth."+key+" off from the console", func(doc *settings.Doc) error {
|
||||
doc.Unset([]string{"auth", key})
|
||||
return nil
|
||||
})
|
||||
return actionDoneMsg{title: title, text: said, err: err}
|
||||
}
|
||||
}}
|
||||
}
|
||||
|
||||
// changeClusterSettings is changeSettings for the console: the newest
|
||||
// settings read, changed, and put in force, said in a sentence.
|
||||
func changeClusterSettings(ctx context.Context, c *ctlClient, note string, change func(doc *settings.Doc) error) (string, error) {
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return "", err
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := change(doc); err != nil {
|
||||
return "", err
|
||||
}
|
||||
body, err := doc.Bytes()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
before, _ := config.Parse([]byte(cur.Body))
|
||||
after, err := config.Parse(body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
changed := config.Changed(before, after)
|
||||
if len(changed) == 0 {
|
||||
return "Nothing changed.", nil
|
||||
}
|
||||
var v adminapi.SettingsView
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/settings",
|
||||
adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note}, &v); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return fmt.Sprintf("Settings version %d. Every control plane starts again on it within seconds, standbys first; "+
|
||||
"engines keep serving.", v.Version), nil
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// adForm is the directory form's values, in its order.
|
||||
func adForm(password, ca, allowed, teams string) []string {
|
||||
return []string{"ldaps://dc1.corp.example, ldaps://dc2.corp.example", "OU=Staff,DC=corp,DC=example",
|
||||
"CN=svc,OU=Service,DC=corp,DC=example", password, ca, allowed, "llmgw-admins", "", teams, "Acme", "yes"}
|
||||
}
|
||||
|
||||
func TestTheDirectoryFormMakesTheBlockTheSettingsKeep(t *testing.T) {
|
||||
pem := "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n"
|
||||
caFile := filepath.Join(t.TempDir(), "ca.pem")
|
||||
if err := os.WriteFile(caFile, []byte(pem), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, err := directoryFromForm(config.LDAP{}, adForm("pw", caFile, "CN=llmgw-users,OU=Groups,DC=corp,DC=example",
|
||||
"ml-team=team:Acme/ml; sales=org:Sales"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The group, nested ones counting; the CA written into the settings;
|
||||
// the password named in the store, not a file on one machine.
|
||||
if l.UserFilter != "(memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)" ||
|
||||
l.CA != pem || l.BindPasswordFile != "store:auth/ldap-bind" || len(l.URLs) != 2 ||
|
||||
l.Memberships["sales"] != "org:Sales" || l.Roles["admin"][0] != "llmgw-admins" || l.Roles["operator"] != nil {
|
||||
t.Errorf("%+v", l)
|
||||
}
|
||||
// What the settings keep reads back the same.
|
||||
back, err := config.ParseLDAP([]byte(ldapYAML(l)))
|
||||
if err != nil || back.UserFilter != l.UserFilter || back.CA != pem || back.Memberships["ml-team"] != "team:Acme/ml" {
|
||||
t.Errorf("read back %+v %v", back, err)
|
||||
}
|
||||
|
||||
// Changed again with nothing typed for the password or the CA: both kept.
|
||||
again, err := directoryFromForm(l, adForm("", "", "(objectClass=user)", ""))
|
||||
if err != nil || again.CA != pem || again.BindPasswordFile != l.BindPasswordFile || again.UserFilter != "(objectClass=user)" ||
|
||||
again.Memberships != nil {
|
||||
t.Errorf("kept: %+v %v", again, err)
|
||||
}
|
||||
// A group by its bare name cannot be a filter, and a plaintext server is refused.
|
||||
if _, err := directoryFromForm(config.LDAP{}, adForm("pw", "", "llmgw-users", "")); err == nil {
|
||||
t.Error("a bare group name was taken")
|
||||
}
|
||||
plain := adForm("pw", "", "", "")
|
||||
plain[0] = "ldap://dc1"
|
||||
if _, err := directoryFromForm(config.LDAP{}, plain); err == nil || !strings.Contains(err.Error(), "plaintext") {
|
||||
t.Errorf("plaintext: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheEntraFormMakesTheIssuerFromTheTenant(t *testing.T) {
|
||||
o, err := oidcFromForm(config.OIDC{}, []string{"0f2c-tenant", "app-id", "s3cret", "g-admins", "", "Microsoft", "yes", "no"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if o.Issuer != "https://login.microsoftonline.com/0f2c-tenant/v2.0" || o.ClientSecretFile != "store:auth/oidc-client-secret" ||
|
||||
o.Roles["admin"][0] != "g-admins" || !o.CreateAccounts || o.NoPasswords {
|
||||
t.Errorf("%+v", o)
|
||||
}
|
||||
// Another provider by its issuer; the secret kept when none is typed.
|
||||
k, err := oidcFromForm(o, []string{"https://sso.example/realms/acme", "llmgw", "", "", "", "Acme SSO", "no", "yes"})
|
||||
if err != nil || k.Issuer != "https://sso.example/realms/acme" || k.ClientSecretFile != o.ClientSecretFile || !k.NoPasswords {
|
||||
t.Errorf("%+v %v", k, err)
|
||||
}
|
||||
if _, err := oidcFromForm(config.OIDC{}, []string{"t", "app", "", "", "", "", "yes", "no"}); err == nil {
|
||||
t.Error("a first setup without a secret was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSignInPanelSaysWhatIsOnAndHowTheDirectoryLastWent(t *testing.T) {
|
||||
m := consoleModel{menu: cSignIn, loaded: true, width: 120, height: 40}
|
||||
m.signIn = signInState{loaded: true, view: adminapi.AuthView{
|
||||
BaseURL: "https://llm.example",
|
||||
OIDC: &adminapi.OIDCView{Issuer: "https://login.microsoftonline.com/t/v2.0", Label: "Microsoft", CreateAccounts: true},
|
||||
LDAP: &adminapi.LDAPView{URLs: []string{"ldaps://dc1"}, BaseDN: "OU=Staff,DC=corp",
|
||||
LastSync: &adminapi.DirectorySyncView{At: time.Now().Add(-time.Minute), People: 40, Accounts: 12, Suspended: 1}},
|
||||
}}
|
||||
_, body, _ := m.signInPanel(110)
|
||||
for _, want := range []string{"Single sign-on (Entra ID, OIDC)", "Microsoft", "https://llm.example/login/oidc/callback",
|
||||
"Active Directory (LDAP)", "ldaps://dc1", "40 people there, 12 accounts kept in step, 1 suspended", "Provisioning (SCIM)"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the panel lacks %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
m.signIn.view.LDAP.Off = "auth.ldap.ca: holds no certificate"
|
||||
if _, body, _ := m.signInPanel(110); !strings.Contains(body, "broken here") || !strings.Contains(body, "holds no certificate") {
|
||||
t.Errorf("a directory off here is not said:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAuthBlocksAreReadFromTheSettingsDocument(t *testing.T) {
|
||||
body := "auth:\n ldap:\n urls: [ldaps://dc1]\n base_dn: DC=corp\n bind_dn: CN=svc\n bind_password_file: store:auth/ldap-bind\n" +
|
||||
" sync_interval: 1m\n oidc:\n issuer: https://sso.example\n client_id: llmgw\n"
|
||||
l, o := authBlocks(body)
|
||||
if l.BaseDN != "DC=corp" || l.SyncInterval != time.Minute || o.ClientID != "llmgw" {
|
||||
t.Errorf("%+v %+v", l, o)
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,9 @@ import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/knadh/koanf/parsers/yaml"
|
||||
"github.com/knadh/koanf/v2"
|
||||
)
|
||||
|
||||
// LDAP is sign-in against a directory — Active Directory above all — with
|
||||
@@ -38,6 +41,10 @@ type LDAP struct {
|
||||
// CAFile is the authority the servers' certificates are checked against,
|
||||
// when it is not the system's: an AD Certificate Services root.
|
||||
CAFile string `koanf:"ca_file"`
|
||||
// CA is the same authority written out, PEM: kept in the cluster's
|
||||
// settings, so no file has to be put on every control plane. Used in
|
||||
// place of CAFile when both are set.
|
||||
CA string `koanf:"ca"`
|
||||
// BindDN and the file holding its password: a service account that may
|
||||
// read users and groups. Nothing is written to the directory.
|
||||
BindDN string `koanf:"bind_dn"`
|
||||
@@ -99,6 +106,8 @@ func (l LDAP) Validate() error {
|
||||
switch {
|
||||
case strings.TrimSpace(l.BindDN) == "" || l.BindPasswordFile == "":
|
||||
return errors.New("auth.ldap: bind_dn and bind_password_file are needed, for the account that finds people")
|
||||
case l.CA != "" && !strings.Contains(l.CA, "-----BEGIN CERTIFICATE-----"):
|
||||
return errors.New("auth.ldap.ca: not a PEM certificate")
|
||||
case strings.TrimSpace(l.BaseDN) == "":
|
||||
return errors.New("auth.ldap: base_dn is needed, where people are searched for")
|
||||
case l.UserFilter != "" && !(strings.HasPrefix(l.UserFilter, "(") && strings.HasSuffix(l.UserFilter, ")")):
|
||||
@@ -135,3 +144,17 @@ func (l LDAP) RoundTrip() time.Duration {
|
||||
}
|
||||
return l.Timeout
|
||||
}
|
||||
|
||||
// ParseLDAP is an auth.ldap block, as YAML, read on its own as the cluster's
|
||||
// settings would read it, and checked.
|
||||
func ParseLDAP(block []byte) (LDAP, error) {
|
||||
var l LDAP
|
||||
k := koanf.New(".")
|
||||
if err := k.Load(bytesProvider(block), yaml.Parser()); err != nil {
|
||||
return l, fmt.Errorf("auth.ldap: %w", err)
|
||||
}
|
||||
if err := k.UnmarshalWithConf("", &l, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
|
||||
return l, fmt.Errorf("auth.ldap: %w", err)
|
||||
}
|
||||
return l, l.Validate()
|
||||
}
|
||||
@@ -76,6 +76,7 @@ type API struct {
|
||||
// train is adapter training; nil without a store and agents.
|
||||
train *TrainAdmin
|
||||
settings *Settings
|
||||
auth *AuthAdmin
|
||||
// restartAgents tells every connected agent to start again.
|
||||
restartAgents func(reason string) []string
|
||||
// revoke records a removed node, so its certificate is refused.
|
||||
@@ -129,6 +130,7 @@ func (a *API) Register(mux Mux) {
|
||||
a.registerTrain(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
a.registerAuth(mux)
|
||||
mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
|
||||
mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
|
||||
mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AuthAdmin is what the sign-in endpoints need: how people sign in now, a
|
||||
// directory's settings tried before they are kept, and where the secrets
|
||||
// those settings name are kept for every control plane.
|
||||
type AuthAdmin struct {
|
||||
Status func() AuthView
|
||||
// TestLDAP tries an auth.ldap block, as YAML, binding with password
|
||||
// when one is given rather than the one the block names.
|
||||
TestLDAP func(ctx context.Context, settings, password string) ([]CheckView, error)
|
||||
Secrets interface {
|
||||
SetClusterSecret(ctx context.Context, name, value string) error
|
||||
}
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
// SetAuth enables the sign-in endpoints.
|
||||
func (a *API) SetAuth(x *AuthAdmin) { a.auth = x }
|
||||
|
||||
// AuthView is how people sign in on this gateway.
|
||||
type AuthView struct {
|
||||
SignupOpen bool `json:"signup_open"`
|
||||
// OIDC is single sign-on through an identity provider: Entra ID,
|
||||
// Keycloak, Okta.
|
||||
OIDC *OIDCView `json:"oidc,omitempty"`
|
||||
// LDAP is sign-in checked by a directory: Active Directory.
|
||||
LDAP *LDAPView `json:"ldap,omitempty"`
|
||||
// SCIM is whether a provider may keep accounts in step.
|
||||
SCIM bool `json:"scim"`
|
||||
// BaseURL is the portal's address, when the settings say it: what a
|
||||
// provider's redirect address is made from.
|
||||
BaseURL string `json:"base_url,omitempty"`
|
||||
}
|
||||
|
||||
// OIDCView is auth.oidc as it is in force.
|
||||
type OIDCView struct {
|
||||
Issuer string `json:"issuer"`
|
||||
ClientID string `json:"client_id"`
|
||||
Label string `json:"label,omitempty"`
|
||||
CreateAccounts bool `json:"create_accounts"`
|
||||
NoPasswords bool `json:"no_passwords"`
|
||||
}
|
||||
|
||||
// LDAPView is auth.ldap as it is in force, and how its last sync went.
|
||||
type LDAPView struct {
|
||||
URLs []string `json:"urls"`
|
||||
BaseDN string `json:"base_dn"`
|
||||
BindDN string `json:"bind_dn"`
|
||||
Label string `json:"label,omitempty"`
|
||||
// Off is why this control plane could not use the settings, when it
|
||||
// could not; local accounts still sign in.
|
||||
Off string `json:"off,omitempty"`
|
||||
LastSync *DirectorySyncView `json:"last_sync,omitempty"`
|
||||
}
|
||||
|
||||
// DirectorySyncView is the last time the leader read the directory.
|
||||
type DirectorySyncView struct {
|
||||
At time.Time `json:"at"`
|
||||
People int `json:"people"`
|
||||
Accounts int `json:"accounts"`
|
||||
Suspended int `json:"suspended"`
|
||||
Reinstated int `json:"reinstated"`
|
||||
RolesChanged int `json:"roles_changed"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// CheckView is one step of trying a directory.
|
||||
type CheckView struct {
|
||||
Step string `json:"step"`
|
||||
OK bool `json:"ok"`
|
||||
Detail string `json:"detail"`
|
||||
}
|
||||
|
||||
// LDAPTestRequest is an auth.ldap block to try, as YAML, and the service
|
||||
// account's password when it is not kept yet.
|
||||
type LDAPTestRequest struct {
|
||||
Settings string `json:"settings"`
|
||||
Password string `json:"password,omitempty"`
|
||||
}
|
||||
|
||||
// LDAPTestView is how trying the directory went.
|
||||
type LDAPTestView struct {
|
||||
OK bool `json:"ok"`
|
||||
Checks []CheckView `json:"checks"`
|
||||
}
|
||||
|
||||
// SecretRequest is a secret for the store to keep.
|
||||
type SecretRequest struct {
|
||||
Value string `json:"value"`
|
||||
}
|
||||
|
||||
// authSecrets are the secrets an operator may give through the API, and
|
||||
// the reference the settings then name each by.
|
||||
var authSecrets = map[string]bool{"ldap-bind": true, "oidc-client-secret": true, "scim-token": true}
|
||||
|
||||
// AuthSecretRef is the reference auth's settings name a kept secret by.
|
||||
func AuthSecretRef(name string) string { return "store:auth/" + name }
|
||||
|
||||
func (a *API) registerAuth(mux Mux) {
|
||||
need := func(w http.ResponseWriter) bool {
|
||||
if a.auth == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no accounts")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
mux.HandleFunc("GET /api/v1/auth", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.auth.Status())
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/auth/ldap/test", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
var req LDAPTestRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
checks, err := a.auth.TestLDAP(r.Context(), req.Settings, req.Password)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
v := LDAPTestView{OK: len(checks) > 0, Checks: checks}
|
||||
for _, c := range checks {
|
||||
v.OK = v.OK && c.OK
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/auth/secrets/{name}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
name := r.PathValue("name")
|
||||
if !authSecrets[name] {
|
||||
writeError(w, http.StatusNotFound, "no such secret: ldap-bind, oidc-client-secret or scim-token")
|
||||
return
|
||||
}
|
||||
var req SecretRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.Value) == "" {
|
||||
writeError(w, http.StatusBadRequest, "an empty secret")
|
||||
return
|
||||
}
|
||||
if err := a.auth.Secrets.SetClusterSecret(r.Context(), "auth/"+name, req.Value); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
if a.auth.Log != nil {
|
||||
// What changed and who changed it; never the value.
|
||||
a.auth.Log.Info("a sign-in secret was set", "secret", name, "by", actorOf(r))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"ref": AuthSecretRef(name)})
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type memSecrets map[string]string
|
||||
|
||||
func (m memSecrets) SetClusterSecret(_ context.Context, name, value string) error {
|
||||
m[name] = value
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestSignInIsShownTriedAndItsSecretsKept(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
kept := memSecrets{}
|
||||
var tried string
|
||||
a.SetAuth(&AuthAdmin{
|
||||
Status: func() AuthView { return AuthView{LDAP: &LDAPView{URLs: []string{"ldaps://dc1"}}} },
|
||||
TestLDAP: func(_ context.Context, settings, password string) ([]CheckView, error) {
|
||||
tried = settings + "|" + password
|
||||
return []CheckView{{Step: "ldaps://dc1", OK: true}, {Step: "people", OK: false, Detail: "no one"}}, nil
|
||||
},
|
||||
Secrets: kept,
|
||||
})
|
||||
var v AuthView
|
||||
if code := call(t, a, "GET", "/api/v1/auth", "", &v); code != 200 || v.LDAP == nil {
|
||||
t.Fatalf("status: %d %+v", code, v)
|
||||
}
|
||||
// One step failing fails the whole.
|
||||
var tv LDAPTestView
|
||||
if code := call(t, a, "POST", "/api/v1/auth/ldap/test", `{"settings":"urls: [ldaps://dc1]","password":"pw"}`, &tv); code != 200 ||
|
||||
tv.OK || len(tv.Checks) != 2 || tried != "urls: [ldaps://dc1]|pw" {
|
||||
t.Errorf("test: %d %+v %q", code, tv, tried)
|
||||
}
|
||||
var ref map[string]string
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":"s3cret"}`, &ref); code != 200 ||
|
||||
kept["auth/ldap-bind"] != "s3cret" || ref["ref"] != "store:auth/ldap-bind" {
|
||||
t.Errorf("secret: %d %v %v", code, kept, ref)
|
||||
}
|
||||
// Only the sign-in secrets, and never an empty one.
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/engine-key", `{"value":"x"}`, nil); code != 404 {
|
||||
t.Errorf("another secret: %d", code)
|
||||
}
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":" "}`, nil); code != 400 {
|
||||
t.Errorf("an empty secret: %d", code)
|
||||
}
|
||||
if strings.Contains(kept["auth/ldap-bind"], " ") {
|
||||
t.Error("an empty secret replaced the kept one")
|
||||
}
|
||||
}
|
||||
@@ -67,6 +67,7 @@ var PathParams = map[string]string{
|
||||
"models/{name}": "The model's name in the catalog.",
|
||||
"presets/{name}": "The preset's name, as `llmgw ctl model presets` lists it.",
|
||||
"nodes/{id}": "The node's ID, as `llmgw ctl nodes` lists it.",
|
||||
"secrets/{name}": "ldap-bind, oidc-client-secret or scim-token.",
|
||||
"jobs/{id}": "The job's ID, as starting it answered.",
|
||||
"runs/{id}": "The quality run's ID, as starting it answered.",
|
||||
"requests/{id}": "The request's ID: the x-request-id a client was answered with.",
|
||||
@@ -396,6 +397,15 @@ var Routes = []Route{
|
||||
{Method: "POST", Path: "/api/v1/settings/rollback", Summary: "Put an earlier version of the cluster's settings back in force",
|
||||
Request: SettingsRollbackRequest{}, Response: SettingsView{}},
|
||||
|
||||
// How people sign in: single sign-on, a directory, and the secrets
|
||||
// they need kept for every control plane.
|
||||
{Method: "GET", Path: "/api/v1/auth", Summary: "How people sign in: OIDC, a directory, SCIM, and the directory's last sync",
|
||||
Response: AuthView{}},
|
||||
{Method: "POST", Path: "/api/v1/auth/ldap/test", Summary: "Try an auth.ldap block before it is kept: servers, the service account, people and groups",
|
||||
Request: LDAPTestRequest{}, Response: LDAPTestView{}},
|
||||
{Method: "PUT", Path: "/api/v1/auth/secrets/{name}", Summary: "Keep ldap-bind, oidc-client-secret or scim-token in the store, for settings to name as store:auth/NAME",
|
||||
Request: SecretRequest{}, Response: map[string]string{}},
|
||||
|
||||
// Usage, requests and the record of changes.
|
||||
{Method: "GET", Path: "/api/v1/usage", Summary: "What each group of the tenancy tree spent",
|
||||
Query: []Param{
|
||||
|
||||
+138
-11
@@ -5,6 +5,7 @@ import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/binary"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
@@ -24,6 +25,26 @@ type LDAP struct {
|
||||
cfg config.LDAP
|
||||
tls *tls.Config
|
||||
dial func(url string) (ldapConn, error)
|
||||
// password, when set, is the service account's in place of the one
|
||||
// bind_password_file names: settings tried before they are kept.
|
||||
password string
|
||||
}
|
||||
|
||||
// WithPassword has the service account bind with pw rather than read it.
|
||||
func (d *LDAP) WithPassword(pw string) *LDAP {
|
||||
d.password = pw
|
||||
return d
|
||||
}
|
||||
|
||||
func (d *LDAP) servicePassword(ctx context.Context) (string, error) {
|
||||
if d.password != "" {
|
||||
return d.password, nil
|
||||
}
|
||||
pw, err := secrets.Read(ctx, d.cfg.BindPasswordFile)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("auth.ldap.bind_password_file: %w", err)
|
||||
}
|
||||
return strings.TrimSpace(string(pw)), nil
|
||||
}
|
||||
|
||||
// ldapConn is the part of a connection used here, so tests need no server.
|
||||
@@ -49,14 +70,18 @@ var ErrBadCredentials = errors.New("the directory does not take that name and pa
|
||||
// NewLDAP prepares a directory from its configuration.
|
||||
func NewLDAP(cfg config.LDAP) (*LDAP, error) {
|
||||
tc := &tls.Config{MinVersion: tls.VersionTLS12}
|
||||
if cfg.CAFile != "" {
|
||||
pem, err := os.ReadFile(cfg.CAFile)
|
||||
if err != nil {
|
||||
ca, where := []byte(cfg.CA), "auth.ldap.ca"
|
||||
if cfg.CA == "" && cfg.CAFile != "" {
|
||||
var err error
|
||||
if ca, err = os.ReadFile(cfg.CAFile); err != nil {
|
||||
return nil, fmt.Errorf("auth.ldap.ca_file: %w", err)
|
||||
}
|
||||
pool := x509.NewCertPool()
|
||||
if !pool.AppendCertsFromPEM(pem) {
|
||||
return nil, fmt.Errorf("auth.ldap.ca_file %s holds no certificate", cfg.CAFile)
|
||||
where = "auth.ldap.ca_file " + cfg.CAFile
|
||||
}
|
||||
if len(ca) > 0 {
|
||||
pool, err := certPool(ca)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", where, err)
|
||||
}
|
||||
tc.RootCAs = pool
|
||||
}
|
||||
@@ -65,6 +90,33 @@ func NewLDAP(cfg config.LDAP) (*LDAP, error) {
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// certPool is the certificates in pem, each of which must parse: a
|
||||
// certificate Go refuses (a negative serial, as Samba's own has) is said,
|
||||
// not skipped into an empty pool.
|
||||
func certPool(data []byte) (*x509.CertPool, error) {
|
||||
pool := x509.NewCertPool()
|
||||
n := 0
|
||||
for rest := data; ; {
|
||||
var b *pem.Block
|
||||
if b, rest = pem.Decode(rest); b == nil {
|
||||
break
|
||||
}
|
||||
if b.Type != "CERTIFICATE" {
|
||||
continue
|
||||
}
|
||||
c, err := x509.ParseCertificate(b.Bytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pool.AddCert(c)
|
||||
n++
|
||||
}
|
||||
if n == 0 {
|
||||
return nil, errors.New("holds no certificate")
|
||||
}
|
||||
return pool, nil
|
||||
}
|
||||
|
||||
func (d *LDAP) dialURL(u string) (ldapConn, error) {
|
||||
c, err := ldap.DialURL(u, ldap.DialWithTLSConfig(d.tls))
|
||||
if err != nil {
|
||||
@@ -84,9 +136,9 @@ func (d *LDAP) dialURL(u string) (ldapConn, error) {
|
||||
// service is a connection bound as the account that finds people, to the
|
||||
// first server that answers.
|
||||
func (d *LDAP) service(ctx context.Context) (ldapConn, error) {
|
||||
pw, err := secrets.Read(ctx, d.cfg.BindPasswordFile)
|
||||
pw, err := d.servicePassword(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("auth.ldap.bind_password_file: %w", err)
|
||||
return nil, err
|
||||
}
|
||||
var errs []error
|
||||
for _, u := range d.cfg.URLs {
|
||||
@@ -95,7 +147,7 @@ func (d *LDAP) service(ctx context.Context) (ldapConn, error) {
|
||||
errs = append(errs, fmt.Errorf("%s: %w", u, err))
|
||||
continue
|
||||
}
|
||||
if err := c.Bind(d.cfg.BindDN, strings.TrimSpace(string(pw))); err != nil {
|
||||
if err := c.Bind(d.cfg.BindDN, pw); err != nil {
|
||||
c.Close()
|
||||
// A server that refuses the service account refuses it
|
||||
// everywhere: not tried on the next.
|
||||
@@ -144,11 +196,11 @@ func (d *LDAP) Authenticate(ctx context.Context, login, password string) (Identi
|
||||
return Identity{}, fmt.Errorf("checking %s's password: %w", login, err)
|
||||
}
|
||||
// Back to the service account to read groups, which a person may not.
|
||||
pw, err := secrets.Read(ctx, d.cfg.BindPasswordFile)
|
||||
pw, err := d.servicePassword(ctx)
|
||||
if err != nil {
|
||||
return Identity{}, err
|
||||
}
|
||||
if err := c.Bind(d.cfg.BindDN, strings.TrimSpace(string(pw))); err != nil {
|
||||
if err := c.Bind(d.cfg.BindDN, pw); err != nil {
|
||||
return Identity{}, err
|
||||
}
|
||||
if id.Groups, err = d.groupsOf(c, id.DN); err != nil {
|
||||
@@ -216,6 +268,81 @@ func (d *LDAP) People(ctx context.Context, m Mapping) ([]Identity, error) {
|
||||
return people, nil
|
||||
}
|
||||
|
||||
// Check is one step of trying a directory.
|
||||
type Check struct {
|
||||
Step string `json:"step"`
|
||||
OK bool `json:"ok"`
|
||||
Detail string `json:"detail"`
|
||||
}
|
||||
|
||||
// Check tries the directory as configured, before its settings are kept:
|
||||
// each server reached and the service account bound on it, how many people
|
||||
// may sign in, and how many are in each group the mapping names.
|
||||
func (d *LDAP) Check(ctx context.Context, m Mapping) []Check {
|
||||
pw, err := d.servicePassword(ctx)
|
||||
if err != nil {
|
||||
return []Check{{Step: "password", Detail: err.Error()}}
|
||||
}
|
||||
var out []Check
|
||||
var c ldapConn
|
||||
for _, u := range d.cfg.URLs {
|
||||
conn, err := d.dial(u)
|
||||
if err == nil {
|
||||
if err = conn.Bind(d.cfg.BindDN, pw); err != nil {
|
||||
conn.Close()
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
out = append(out, Check{Step: u, Detail: err.Error()})
|
||||
continue
|
||||
}
|
||||
out = append(out, Check{Step: u, OK: true, Detail: "reached, bound as " + d.cfg.BindDN})
|
||||
if c == nil {
|
||||
c = conn
|
||||
} else {
|
||||
conn.Close()
|
||||
}
|
||||
}
|
||||
if c == nil {
|
||||
return out
|
||||
}
|
||||
defer c.Close()
|
||||
res, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
|
||||
0, 0, false, d.people(false), []string{"objectGUID"}, nil), 500)
|
||||
switch {
|
||||
case err != nil:
|
||||
out = append(out, Check{Step: "people", Detail: err.Error()})
|
||||
case len(res.Entries) == 0:
|
||||
out = append(out, Check{Step: "people", Detail: "no enabled user under base_dn matches user_filter: no one could sign in"})
|
||||
default:
|
||||
out = append(out, Check{Step: "people", OK: true,
|
||||
Detail: fmt.Sprintf("%d enabled people under base_dn may sign in", len(res.Entries))})
|
||||
}
|
||||
for _, g := range m.Groups() {
|
||||
dns, err := d.groupDNs(c, g)
|
||||
if err != nil || len(dns) == 0 {
|
||||
detail := "no group of that name under " + d.groupBase()
|
||||
if err != nil {
|
||||
detail = err.Error()
|
||||
}
|
||||
out = append(out, Check{Step: "group " + g, Detail: detail})
|
||||
continue
|
||||
}
|
||||
n := 0
|
||||
for _, gdn := range dns {
|
||||
filter := "(&" + d.people(false) + "(memberOf:" + inChain + ":=" + ldap.EscapeFilter(gdn) + "))"
|
||||
r, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree,
|
||||
ldap.NeverDerefAliases, 0, 0, false, filter, []string{"objectGUID"}, nil), 500)
|
||||
if err == nil {
|
||||
n += len(r.Entries)
|
||||
}
|
||||
}
|
||||
out = append(out, Check{Step: "group " + g, OK: true,
|
||||
Detail: fmt.Sprintf("%d of the people who may sign in are in it, nested groups included", n)})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// groupDNs are the groups a mapping's name means: a DN as it is, a CN
|
||||
// looked up.
|
||||
func (d *LDAP) groupDNs(c ldapConn, g string) ([]string, error) {
|
||||
|
||||
@@ -18,12 +18,16 @@ type fakeConn struct {
|
||||
filters []string
|
||||
password string // the person's
|
||||
people int // how many the person search finds
|
||||
refused bool // the service account's password is wrong
|
||||
}
|
||||
|
||||
func (f *fakeConn) Bind(user, password string) error {
|
||||
f.binds = append(f.binds, user)
|
||||
if f.refused && strings.HasPrefix(user, "CN=llmgw") {
|
||||
return ldap.NewError(ldap.LDAPResultInvalidCredentials, errors.New("invalid credentials"))
|
||||
}
|
||||
if strings.HasPrefix(user, "CN=Ada") && password != f.password {
|
||||
return &ldap.Error{ResultCode: ldap.LDAPResultInvalidCredentials}
|
||||
return ldap.NewError(ldap.LDAPResultInvalidCredentials, errors.New("invalid credentials"))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -128,3 +132,31 @@ func slicesHas(xs []string, x string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func TestADirectoryIsTriedStepByStepBeforeItsSettingsAreKept(t *testing.T) {
|
||||
m := Mapping{Roles: map[string][]string{"admin": {"llmgw-admins"}}}
|
||||
checks := fakeLDAP(t, &fakeConn{people: 1}).WithPassword("typed-in").Check(t.Context(), m)
|
||||
if len(checks) != 3 || !checks[0].OK || checks[0].Step != "ldaps://dc1" || !checks[1].OK ||
|
||||
checks[2].Step != "group llmgw-admins" || !checks[2].OK {
|
||||
t.Errorf("checks %+v", checks)
|
||||
}
|
||||
// A refused service account stops at the server, and says why.
|
||||
checks = fakeLDAP(t, &fakeConn{refused: true}).Check(t.Context(), m)
|
||||
if len(checks) != 1 || checks[0].OK || !strings.Contains(checks[0].Detail, "Invalid Credentials") {
|
||||
t.Errorf("refused: %+v", checks)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACertificateGoCannotReadIsSaidNotSkipped(t *testing.T) {
|
||||
if _, err := certPool([]byte("not a certificate")); err == nil {
|
||||
t.Error("an empty pool was taken")
|
||||
}
|
||||
// A CERTIFICATE block Go cannot parse, as it cannot Samba's own (its
|
||||
// serial is negative): an error, not an empty pool.
|
||||
neg := "-----BEGIN CERTIFICATE-----\nMIIBNTCB3KADAgECAgT/////MAoGCCqGSM49BAMCMA8xDTALBgNVBAMTBHRlc3Qw\n" +
|
||||
"HhcNMjYwMTAxMDAwMDAwWhcNMjgwMTAxMDAwMDAwWjAPMQ0wCwYDVQQDEwR0ZXN0\n" +
|
||||
"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\n-----END CERTIFICATE-----\n"
|
||||
if _, err := certPool([]byte(neg)); err == nil {
|
||||
t.Error("a broken certificate made a pool")
|
||||
}
|
||||
}
|
||||
@@ -193,6 +193,7 @@ var Reference = []Block{
|
||||
{Key: "urls", Type: "[]string", Doc: "URLs are the directory's servers, tried in turn: ldaps:// (636), or\nldap:// (389) with StartTLS.", Nested: ""},
|
||||
{Key: "start_tls", Type: "bool", Doc: "StartTLS upgrades an ldap:// connection before anything is sent. A\npassword never crosses a connection without TLS: ldap:// without it\nis refused.", Nested: ""},
|
||||
{Key: "ca_file", Type: "string", Doc: "CAFile is the authority the servers' certificates are checked against,\nwhen it is not the system's: an AD Certificate Services root.", Nested: ""},
|
||||
{Key: "ca", Type: "string", Doc: "CA is the same authority written out, PEM: kept in the cluster's\nsettings, so no file has to be put on every control plane. Used in\nplace of CAFile when both are set.", Nested: ""},
|
||||
{Key: "bind_dn", Type: "string", Doc: "BindDN and the file holding its password: a service account that may\nread users and groups. Nothing is written to the directory.", Nested: ""},
|
||||
{Key: "bind_password_file", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "base_dn", Type: "string", Doc: "BaseDN is where people are searched for, subtree included.", Nested: ""},
|
||||
|
||||
@@ -6,10 +6,12 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/account"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/directory"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -24,6 +26,9 @@ type ldapDirectory struct {
|
||||
every time.Duration
|
||||
label string
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
last *adminapi.DirectorySyncView
|
||||
}
|
||||
|
||||
func newLDAPDirectory(cfg config.LDAP, st *storepkg.DB, log *slog.Logger) (*ldapDirectory, error) {
|
||||
@@ -92,16 +97,26 @@ func (d *ldapDirectory) sync(ctx context.Context, leader func() bool) {
|
||||
func (d *ldapDirectory) syncOnce(ctx context.Context) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
|
||||
defer cancel()
|
||||
last := &adminapi.DirectorySyncView{At: time.Now()}
|
||||
defer func() {
|
||||
d.mu.Lock()
|
||||
d.last = last
|
||||
d.mu.Unlock()
|
||||
}()
|
||||
people, err := d.dir.People(ctx, d.keeper.Mapping)
|
||||
if err != nil {
|
||||
d.log.Warn("could not read the directory; accounts left as they are", "err", err)
|
||||
last.Error = "could not read the directory: " + err.Error()
|
||||
return
|
||||
}
|
||||
last.People = len(people)
|
||||
rep, err := d.keeper.Sync(ctx, people)
|
||||
if err != nil {
|
||||
d.log.Error("the directory sync stopped", "err", err, "people", len(people))
|
||||
last.Error = err.Error()
|
||||
return
|
||||
}
|
||||
last.Accounts, last.Suspended, last.Reinstated, last.RolesChanged = rep.Seen, rep.Suspended, rep.Reinstated, rep.Updated
|
||||
for _, where := range rep.Unmapped {
|
||||
d.log.Warn("auth.ldap.memberships maps a group to a place the tree does not have", "place", where)
|
||||
}
|
||||
@@ -110,3 +125,54 @@ func (d *ldapDirectory) syncOnce(ctx context.Context) {
|
||||
"suspended", rep.Suspended, "reinstated", rep.Reinstated, "roles_changed", rep.Updated)
|
||||
}
|
||||
}
|
||||
|
||||
// lastSync is how the leader's last read of the directory went; nil before
|
||||
// the first, and on a standby.
|
||||
func (d *ldapDirectory) lastSync() *adminapi.DirectorySyncView {
|
||||
d.mu.Lock()
|
||||
defer d.mu.Unlock()
|
||||
return d.last
|
||||
}
|
||||
|
||||
// AuthStatus is how people sign in on this gateway, for the admin API.
|
||||
func (r *Router) AuthStatus() adminapi.AuthView {
|
||||
v := adminapi.AuthView{SCIM: r.scim.TokenFile != "", BaseURL: r.mail.BaseURL}
|
||||
if r.accounts != nil {
|
||||
v.SignupOpen = r.accounts.SignupOpen()
|
||||
}
|
||||
if o := r.oidc; o.Enabled() {
|
||||
v.OIDC = &adminapi.OIDCView{Issuer: o.Issuer, ClientID: o.ClientID, Label: o.Label,
|
||||
CreateAccounts: o.CreateAccounts, NoPasswords: o.NoPasswords}
|
||||
}
|
||||
if l := r.ldap; l.Enabled() {
|
||||
v.LDAP = &adminapi.LDAPView{URLs: l.URLs, BaseDN: l.BaseDN, BindDN: l.BindDN, Label: l.Label, Off: r.ldapOff}
|
||||
if r.directory != nil {
|
||||
v.LDAP.LastSync = r.directory.lastSync()
|
||||
}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// TestDirectory tries an auth.ldap block, given as YAML, before it is kept:
|
||||
// checked as the settings would be, then each step against the directory.
|
||||
func (r *Router) TestDirectory(ctx context.Context, settings, password string) ([]adminapi.CheckView, error) {
|
||||
l, err := config.ParseLDAP([]byte(settings))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
dir, err := directory.NewLDAP(l)
|
||||
if err != nil {
|
||||
return []adminapi.CheckView{{Step: "certificate", Detail: err.Error()}}, nil
|
||||
}
|
||||
if password != "" {
|
||||
dir.WithPassword(password)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, time.Minute)
|
||||
defer cancel()
|
||||
checks := dir.Check(ctx, directory.Mapping{Roles: l.Roles, Memberships: l.Memberships})
|
||||
out := make([]adminapi.CheckView, len(checks))
|
||||
for i, c := range checks {
|
||||
out[i] = adminapi.CheckView(c)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
)
|
||||
|
||||
// auth.ldap is the cluster's: a CA file one machine lacks must not keep its
|
||||
// control plane from starting, where nothing could put the setting right.
|
||||
func TestADirectoryThatCannotBeUsedHereLeavesTheGatewayUp(t *testing.T) {
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: "http://127.0.0.1:1", Models: []string{"m"}})
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.LDAP = config.LDAP{URLs: []string{"ldaps://dc1"}, CAFile: filepath.Join(t.TempDir(), "missing.pem"),
|
||||
BindDN: "CN=svc", BindPasswordFile: "/nonexistent", BaseDN: "DC=corp"}
|
||||
r := newTestRouter(t, cfg)
|
||||
if r.directory != nil {
|
||||
t.Fatal("a directory was set up from a CA file that is not there")
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
r.Handler().ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/login", nil))
|
||||
if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), `type="email"`) {
|
||||
t.Errorf("/login: %d", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// Settings tried are checked as kept ones are: a plaintext server is
|
||||
// refused before anything is sent to it.
|
||||
func TestDirectorySettingsTriedAreCheckedFirst(t *testing.T) {
|
||||
r := &Router{}
|
||||
if _, err := r.TestDirectory(t.Context(), "urls: [ldap://dc1]\nbind_dn: CN=svc\nbind_password_file: store:auth/ldap-bind\nbase_dn: DC=corp\n", "pw"); err == nil ||
|
||||
!strings.Contains(err.Error(), "plaintext") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
checks, err := r.TestDirectory(t.Context(), "urls: [ldaps://dc1]\nca: \"-----BEGIN CERTIFICATE-----\\nAAAA\\n-----END CERTIFICATE-----\"\n"+
|
||||
"bind_dn: CN=svc\nbind_password_file: store:auth/ldap-bind\nbase_dn: DC=corp\n", "pw")
|
||||
if err != nil || len(checks) != 1 || checks[0].Step != "certificate" || checks[0].OK {
|
||||
t.Errorf("a broken CA: %+v %v", checks, err)
|
||||
}
|
||||
}
|
||||
@@ -43,6 +43,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/scheduler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/scrape"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
@@ -120,7 +121,11 @@ type Router struct {
|
||||
// notifier says what the operator must act on; nil says nothing.
|
||||
notifier *notify.Notifier
|
||||
// directory is auth.ldap's: sign-in and the accounts kept in step.
|
||||
// ldap is the setting, and ldapOff why this control plane could not
|
||||
// use it.
|
||||
directory *ldapDirectory
|
||||
ldap config.LDAP
|
||||
ldapOff string
|
||||
// guard judges requests and answers; nil judges nothing. guardLibrary
|
||||
// is the policies it was built from, for the pages that show them.
|
||||
guard guard.Guard
|
||||
@@ -358,6 +363,9 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
return nil, fmt.Errorf("auth store: %w", err)
|
||||
}
|
||||
r.store = st
|
||||
// Secrets the settings name as store:… are the same on every
|
||||
// control plane: read from here.
|
||||
secrets.SetStore(st.StoredSecret)
|
||||
r.usageRetention = cfg.Auth.UsageRetention
|
||||
// Applied where no scope set one, so a project created without limits
|
||||
// is still metered. Zero leaves it unlimited, which was the old
|
||||
@@ -398,12 +406,18 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
SignupOpen: cfg.Auth.SignupOpen,
|
||||
}, log)
|
||||
if cfg.Auth.LDAP.Enabled() {
|
||||
d, err := newLDAPDirectory(cfg.Auth.LDAP, st, log)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
// A setting every control plane shares: one that cannot be used
|
||||
// here (a CA file missing on this machine) must not keep them all
|
||||
// down, where nothing could be used to put it right. Local
|
||||
// accounts still sign in.
|
||||
r.ldap = cfg.Auth.LDAP
|
||||
if d, err := newLDAPDirectory(cfg.Auth.LDAP, st, log); err != nil {
|
||||
log.Error("sign-in with the directory is off on this control plane", "err", err)
|
||||
r.ldapOff = err.Error()
|
||||
} else {
|
||||
r.directory = d
|
||||
r.accounts.SetDirectory(d)
|
||||
}
|
||||
r.directory = d
|
||||
r.accounts.SetDirectory(d)
|
||||
}
|
||||
|
||||
r.acct = accounting.New(
|
||||
|
||||
@@ -8,6 +8,8 @@
|
||||
// /etc/llmgw/secrets/store-dsn a file, trimmed
|
||||
// env:LLMGW_STORE_DSN an environment variable
|
||||
// vault:secret/llmgw/store#dsn a field of a KV secret in Vault
|
||||
// store:auth/ldap-bind a secret the cluster's store keeps,
|
||||
// on a control plane only
|
||||
//
|
||||
// A Vault path's first segment is its KV v2 mount; secret/data/… is taken
|
||||
// as written. KV v1 mounts work too.
|
||||
@@ -85,7 +87,21 @@ func Configure(v Vault) error {
|
||||
|
||||
// IsReference says whether s names a secret somewhere other than a file.
|
||||
func IsReference(s string) bool {
|
||||
return strings.HasPrefix(s, "env:") || strings.HasPrefix(s, "vault:")
|
||||
return strings.HasPrefix(s, "env:") || strings.HasPrefix(s, "vault:") || strings.HasPrefix(s, "store:")
|
||||
}
|
||||
|
||||
var (
|
||||
storeMu sync.RWMutex
|
||||
storeRead func(ctx context.Context, name string) (string, error)
|
||||
)
|
||||
|
||||
// SetStore is how store: references are read: a control plane sets it once
|
||||
// its store is open. A secret kept there is the same for every control
|
||||
// plane, where a file would have to be put on each.
|
||||
func SetStore(read func(ctx context.Context, name string) (string, error)) {
|
||||
storeMu.Lock()
|
||||
defer storeMu.Unlock()
|
||||
storeRead = read
|
||||
}
|
||||
|
||||
// Read reads a secret by reference, trimmed of surrounding space.
|
||||
@@ -100,6 +116,18 @@ func Read(ctx context.Context, ref string) ([]byte, error) {
|
||||
return []byte(strings.TrimSpace(v)), nil
|
||||
case strings.HasPrefix(ref, "vault:"):
|
||||
return readVault(ctx, strings.TrimPrefix(ref, "vault:"))
|
||||
case strings.HasPrefix(ref, "store:"):
|
||||
storeMu.RLock()
|
||||
read := storeRead
|
||||
storeMu.RUnlock()
|
||||
if read == nil {
|
||||
return nil, fmt.Errorf("%s: a store: secret is read on a control plane, which has the store", ref)
|
||||
}
|
||||
v, err := read(ctx, strings.TrimPrefix(ref, "store:"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", ref, err)
|
||||
}
|
||||
return []byte(strings.TrimSpace(v)), nil
|
||||
}
|
||||
b, err := os.ReadFile(ref) //nolint:gosec // a path the operator configured
|
||||
if err != nil {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package secrets
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
@@ -140,3 +141,15 @@ func TestSealersSealAndOpen(t *testing.T) {
|
||||
t.Errorf("transit open: %q %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStoreReferenceIsReadFromTheStoreOnAControlPlaneOnly(t *testing.T) {
|
||||
SetStore(nil)
|
||||
if _, err := Read(t.Context(), "store:auth/ldap-bind"); err == nil || !strings.Contains(err.Error(), "control plane") {
|
||||
t.Errorf("without a store: %v", err)
|
||||
}
|
||||
SetStore(func(_ context.Context, name string) (string, error) { return " kept:" + name + "\n", nil })
|
||||
t.Cleanup(func() { SetStore(nil) })
|
||||
if b, err := Read(t.Context(), "store:auth/ldap-bind"); err != nil || string(b) != "kept:auth/ldap-bind" {
|
||||
t.Errorf("%q %v", b, err)
|
||||
}
|
||||
}
|
||||
@@ -3,10 +3,13 @@ package store
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
|
||||
)
|
||||
|
||||
// ClusterSecret returns the named secret, creating a random one the first time
|
||||
@@ -39,6 +42,43 @@ func (s *DB) ClusterSecret(ctx context.Context, name string) (string, error) {
|
||||
`SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return s.openSecret(ctx, name, v)
|
||||
}
|
||||
|
||||
// SetClusterSecret keeps value under name, sealed when a master key is set,
|
||||
// in place of what was there: a secret an operator gives, such as the
|
||||
// password the directory is read with, the same for every control plane.
|
||||
func (s *DB) SetClusterSecret(ctx context.Context, name, value string) error {
|
||||
stored := value
|
||||
if s.sealer != nil {
|
||||
var err error
|
||||
if stored, err = s.sealer.Seal(ctx, []byte(value)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO cluster_secrets (name, value, created_at) VALUES (?, ?, ?)
|
||||
ON CONFLICT (name) DO UPDATE SET value = excluded.value, created_at = excluded.created_at`,
|
||||
name, stored, time.Now().UnixMilli())
|
||||
return err
|
||||
}
|
||||
|
||||
// StoredSecret is the named secret, ErrNotFound when none was kept; unlike
|
||||
// ClusterSecret it never makes one.
|
||||
func (s *DB) StoredSecret(ctx context.Context, name string) (string, error) {
|
||||
var v string
|
||||
err := s.db.QueryRowContext(ctx, `SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return "", ErrNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return s.openSecret(ctx, name, v)
|
||||
}
|
||||
|
||||
// openSecret is a kept value in the clear.
|
||||
func (s *DB) openSecret(ctx context.Context, name, v string) (string, error) {
|
||||
if secrets.IsSealed(v) {
|
||||
if s.sealer == nil {
|
||||
return "", fmt.Errorf("the store's secret %q is sealed, and secrets.master_key is not set to open it", name)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
@@ -23,3 +24,22 @@ func TestAClusterSecretIsMadeOnceAndKept(t *testing.T) {
|
||||
t.Error("two names share a secret")
|
||||
}
|
||||
}
|
||||
|
||||
func TestASecretGivenIsKeptReplacedAndNeverMade(t *testing.T) {
|
||||
db, err := Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if _, err := db.StoredSecret(t.Context(), "auth/ldap-bind"); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("a secret never given: %v", err)
|
||||
}
|
||||
for _, v := range []string{"first", "second"} {
|
||||
if err := db.SetClusterSecret(t.Context(), "auth/ldap-bind", v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, err := db.StoredSecret(t.Context(), "auth/ldap-bind"); err != nil || got != v {
|
||||
t.Errorf("read %q %v, want %q", got, err, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -62,6 +62,8 @@
|
||||
"node.memory": "Memory",
|
||||
"node.cores_f": "%s cores",
|
||||
"node.no_engines": "No engines are placed on this node.",
|
||||
"node.version_differs": "the control plane runs %s",
|
||||
"node.gpu.models": "Models",
|
||||
"node.engines": "Engines: %s",
|
||||
"node.engines_none": "Engines: none installed",
|
||||
"node.model_files": "%d model file(s) on disk",
|
||||
|
||||
@@ -61,6 +61,8 @@
|
||||
"node.memory": "Bellek",
|
||||
"node.cores_f": "%s çekirdek",
|
||||
"node.no_engines": "Bu düğüme yerleştirilmiş motor yok.",
|
||||
"node.version_differs": "control plane %s çalıştırıyor",
|
||||
"node.gpu.models": "Modeller",
|
||||
"node.engines": "Motorlar: %s",
|
||||
"node.engines_none": "Motorlar: hiçbiri kurulu değil",
|
||||
"node.model_files": "diskte %d model dosyası",
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"html/template"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -20,9 +21,13 @@ type nodePageData struct {
|
||||
OnAdminListener bool
|
||||
Node adminapi.NodeView
|
||||
Engines []nodeEngine
|
||||
Charts *nodeCharts
|
||||
HasHistory bool
|
||||
Back, Logs string
|
||||
// GPUModels are the models on each of the node's GPUs, by index.
|
||||
GPUModels map[int][]string
|
||||
// EngineVersions are the installed engines and their versions, sorted.
|
||||
EngineVersions []string
|
||||
Charts *nodeCharts
|
||||
HasHistory bool
|
||||
Back, Logs string
|
||||
// Self is this page's address, for the in-place refresh.
|
||||
Self string
|
||||
Version, Now string
|
||||
@@ -34,6 +39,7 @@ type nodePageData struct {
|
||||
// nodeEngine is one engine running on the node, and its share of it.
|
||||
type nodeEngine struct {
|
||||
Replica, Model, Shard, State string
|
||||
GPUs []int
|
||||
CPUCores float64
|
||||
MemoryBytes int64
|
||||
// CPUShare is its cores as a percentage of the machine, for the bar.
|
||||
@@ -128,7 +134,15 @@ func NodePage(src Source, history History, window time.Duration, onAdminListener
|
||||
continue
|
||||
}
|
||||
e := nodeEngine{Replica: rep.ID, Model: rep.Model, Shard: sh.ID, State: sh.State,
|
||||
CPUCores: sh.CPUCores, MemoryBytes: sh.MemoryBytes}
|
||||
GPUs: sh.GPUs, CPUCores: sh.CPUCores, MemoryBytes: sh.MemoryBytes}
|
||||
for _, g := range sh.GPUs {
|
||||
if d.GPUModels == nil {
|
||||
d.GPUModels = map[int][]string{}
|
||||
}
|
||||
if !slices.Contains(d.GPUModels[g], rep.Model) {
|
||||
d.GPUModels[g] = append(d.GPUModels[g], rep.Model)
|
||||
}
|
||||
}
|
||||
if l := d.Node.Load; l != nil && l.CPUs > 0 {
|
||||
e.CPUShare = sh.CPUCores * 100 / float64(l.CPUs)
|
||||
}
|
||||
@@ -137,6 +151,10 @@ func NodePage(src Source, history History, window time.Duration, onAdminListener
|
||||
}
|
||||
}
|
||||
sort.Slice(d.Engines, func(i, j int) bool { return d.Engines[i].CPUCores > d.Engines[j].CPUCores })
|
||||
for e, v := range d.Node.EngineVersions {
|
||||
d.EngineVersions = append(d.EngineVersions, e+" "+v)
|
||||
}
|
||||
sort.Strings(d.EngineVersions)
|
||||
|
||||
if history != nil {
|
||||
d.Charts = loadNodeCharts(r.Context(), l, history, id, replicas, d.Node.GPUTotal, window, log)
|
||||
|
||||
@@ -270,3 +270,48 @@ func TestTheClusterPageDrawsItsGPUsByClass(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// versionSource is hotSource with an agent a release behind, a model on its
|
||||
// second GPU, and engines with their versions.
|
||||
type versionSource struct{ hotSource }
|
||||
|
||||
func (versionSource) Nodes() []adminapi.NodeView {
|
||||
n := hotSource{}.Nodes()[0]
|
||||
n.AgentVersion = "v0"
|
||||
n.EngineVersions = map[string]string{"llamacpp": "b10991", "comfyui": "0.39.2"}
|
||||
return []adminapi.NodeView{n}
|
||||
}
|
||||
|
||||
func (versionSource) Replicas(string) []adminapi.ReplicaView {
|
||||
r := fakeSource{}.Replicas("")
|
||||
r[0].Shards[0].GPUs = []int{1}
|
||||
return r
|
||||
}
|
||||
|
||||
func TestANodesPageSaysWhatItRunsAndWhichModelIsOnEachGPU(t *testing.T) {
|
||||
_, body := get(t, nodeMux(versionSource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
model := versionSource{}.Replicas("")[0].Model
|
||||
for _, want := range []string{
|
||||
// The page's control plane runs v1: an agent on v0 is marked.
|
||||
`class="bad" title="the control plane runs v1">llmgw v0</span>`,
|
||||
"Engines: comfyui 0.39.2, llamacpp b10991",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// GPU 1 holds the model, GPU 0 nothing.
|
||||
rows := strings.Split(body, "<tr>")
|
||||
var gpu0, gpu1 string
|
||||
for _, row := range rows {
|
||||
switch {
|
||||
case strings.HasPrefix(row, "\n <td class=\"num\">0</td>"):
|
||||
gpu0 = row
|
||||
case strings.HasPrefix(row, "\n <td class=\"num\">1</td>"):
|
||||
gpu1 = row
|
||||
}
|
||||
}
|
||||
if !strings.Contains(gpu1, "<td>"+model+"</td>") || strings.Contains(gpu0, model) {
|
||||
t.Errorf("GPU rows:\n0: %.300s\n1: %.300s", gpu0, gpu1)
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,7 @@
|
||||
<h1>{{.Node.ID}}</h1>
|
||||
<p class="sub"><span class="state state-{{.Node.State}}">{{t (print "dash.state." .Node.State)}}</span>
|
||||
{{with .Node.OS}} · {{.}}{{end}}{{with .Node.MgmtAddr}} · {{.}}{{end}}
|
||||
{{with .Node.AgentVersion}} · <span{{if ne . $.Version}} class="bad" title="{{t "node.version_differs" $.Version}}"{{end}}>llmgw {{.}}</span>{{end}}
|
||||
· {{if .Node.LastSeenAgo}}{{t "node.last_seen" .Node.LastSeenAgo}}{{else}}{{t "node.never_seen"}}{{end}} · <a href="{{.Logs}}">{{t "dash.nodes.logs"}}</a></p>
|
||||
{{if .OnAdminListener}}<nav><a href="{{.Back}}">{{t "node.nav.cluster"}}</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
|
||||
</header>
|
||||
@@ -39,7 +40,7 @@
|
||||
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
|
||||
{{with .Load}}{{if .GPUs}}
|
||||
<h2>{{t "node.gpus"}}</h2>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th>{{t "node.gpu.busy"}}</th><th>{{t "node.gpu.memory_used"}}</th><th class="num">{{t "node.gpu.temp"}}</th><th class="num">{{t "node.gpu.memory_temp"}}</th><th>{{t "node.gpu.power"}}</th><th class="num">{{t "node.gpu.clock"}}</th><th>{{t "node.gpu.held_back"}}</th></tr>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th>{{t "node.gpu.busy"}}</th><th>{{t "node.gpu.memory_used"}}</th><th class="num">{{t "node.gpu.temp"}}</th><th class="num">{{t "node.gpu.memory_temp"}}</th><th>{{t "node.gpu.power"}}</th><th class="num">{{t "node.gpu.clock"}}</th><th>{{t "node.gpu.models"}}</th><th>{{t "node.gpu.held_back"}}</th></tr>
|
||||
{{range .GPUs}}<tr>
|
||||
<td class="num">{{.Index}}</td>
|
||||
{{/* From the last sample; a dash where the driver or an older agent did
|
||||
@@ -52,6 +53,7 @@
|
||||
<td class="num">{{if .SMClockMHz}}{{num .SMClockMHz 0}} MHz{{else}}—{{end}}</td>
|
||||
{{/* The driver's reasons and the wear are named as the driver and the
|
||||
inventory name them, in English. */}}
|
||||
<td>{{with index $.GPUModels .Index}}{{join .}}{{else}}<span class="note">—</span>{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">{{t "node.gpu.nothing"}}</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{t "node.gpu.ecc" (num .ECCUncorrected 0)}}</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{t "node.gpu.pcie" (count .PCIeReplays)}}</span>{{end}}{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{end}}{{end}}
|
||||
@@ -106,16 +108,17 @@
|
||||
|
||||
<h2>{{t "node.runs"}}</h2>
|
||||
{{if .Engines}}
|
||||
<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th>{{t "dash.col.replica"}}</th><th>{{t "dash.col.state"}}</th><th>CPU</th><th class="num">{{t "node.memory"}}</th></tr>
|
||||
<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th>{{t "dash.col.replica"}}</th><th>{{t "dash.col.state"}}</th><th>GPU</th><th>CPU</th><th class="num">{{t "node.memory"}}</th></tr>
|
||||
{{range .Engines}}<tr>
|
||||
<td>{{.Model}}</td>
|
||||
<td><code>{{.Replica}}</code></td>
|
||||
<td class="state state-{{.State}}">{{t (print "dash.state." .State)}}</td>
|
||||
<td>{{if .GPUs}}{{range $i, $g := .GPUs}}{{if $i}}, {{end}}{{$g}}{{end}}{{else}}—{{end}}</td>
|
||||
<td class="load">{{template "minibar" pctbar .CPUShare}}<span class="load-v">{{t "node.cores_f" (num .CPUCores 1)}}</span></td>
|
||||
<td class="num">{{if .MemoryBytes}}{{bytes .MemoryBytes}}{{else}}—{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{else}}<p class="empty">{{t "node.no_engines"}}</p>{{end}}
|
||||
<p class="note">{{if .Node.Engines}}{{t "node.engines" (join .Node.Engines)}}{{else}}{{t "node.engines_none"}}{{end}}{{if .Node.Models}} · {{t "node.model_files" (len .Node.Models)}}{{end}}.</p>
|
||||
<p class="note">{{if .EngineVersions}}{{t "node.engines" (join .EngineVersions)}}{{else if .Node.Engines}}{{t "node.engines" (join .Node.Engines)}}{{else}}{{t "node.engines_none"}}{{end}}{{if .Node.Models}} · {{t "node.model_files" (len .Node.Models)}}{{end}}.</p>
|
||||
</div>
|
||||
|
||||
{{if .HasHistory}}{{with .Charts}}
|
||||
|
||||
Reference in new issue
Block a user