Compare commits
No files matched your search
+57
@@ -761,6 +761,63 @@ portal is not done — the page shows; the console and ctl change.
|
||||
|
||||
---
|
||||
|
||||
## E. Every setting takes effect when it is changed (2026-10-05)
|
||||
|
||||
Asked for after a site visit: an upgrade left server.yaml without guardrails,
|
||||
a node on another subnet stayed lost until firewall_networks was edited by
|
||||
hand, and nothing in server.yaml, agent.yaml or a generated unit applies until
|
||||
somebody restarts the right service on every machine. The rule from here on:
|
||||
**a file holds only what one machine needs to start; everything else is the
|
||||
cluster's, kept in the store, changed through ctl, the console or the web,
|
||||
audited, and in force on every control plane within seconds.**
|
||||
|
||||
### E1. Cluster settings in the store (L) — done 2026-10-05
|
||||
Live on the PoC (140, 150): imported as version 1; a guard change applied on
|
||||
both without a restart; a router change restarted the standby at once and the
|
||||
leader 30s later with /health answering throughout; rollback did the same.
|
||||
- Table `cluster_settings` (versioned YAML document, who and why). The
|
||||
cluster-wide part of server.yaml lives there; server.yaml keeps the
|
||||
per-machine keys only (listeners, TLS paths, store DSN, secrets, tracing,
|
||||
monitoring, mirror.api/tls, backup_dir/key, endpoints).
|
||||
- First start against a store with no document imports the file's cluster
|
||||
keys (and the baseline guard policy when there is none); after that those
|
||||
keys in a file are ignored, with a warning.
|
||||
- Each control plane polls the document's version and its own server.yaml's
|
||||
hash. A change is validated, then applied: `guard` live, the rest by a
|
||||
graceful restart of the process (standbys first, the leader last), which
|
||||
systemd brings back. No "restart now?" anywhere.
|
||||
- `llmgw ctl config show|history|get|set|unset|edit|rollback`; the console's
|
||||
settings screen edits the store for cluster keys; admin API
|
||||
`GET/PUT /api/v1/settings`, `/history`, `/rollback`.
|
||||
- A standby that becomes leader re-reads the catalog, engine choices and
|
||||
facility state from the store (it kept what it read at start).
|
||||
- Retires `appendGuardIfMissing` (server init): the import does it.
|
||||
|
||||
### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
|
||||
`firewall.enabled` in the cluster's settings reaches nodes over the agent
|
||||
stream and control planes through llmgw-firewall-sync.path. Left: the mirror
|
||||
machine and store members, the allowed set from membership, the other files.
|
||||
|
||||
Every machine runs the agent (root) with roles — node, control plane, mirror,
|
||||
store member — and the agent renders and applies what was written once by
|
||||
init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and
|
||||
Patroni configuration. The firewall's allowed set is the cluster's membership
|
||||
(every node, control plane, mirror and store member) plus
|
||||
`firewall.extra` from the store; 7443 stays open (mTLS). A node added or
|
||||
removed changes every machine's rules. Retires `firewall.Admit` and
|
||||
`firewall_networks` in server.yaml (imported into the store).
|
||||
|
||||
### E3. agent.yaml down to bootstrap (M)
|
||||
agent.yaml keeps the control planes' addresses, its certificate and its id.
|
||||
Labels, fabrics, engines, heartbeat and supervisor settings move to the
|
||||
store per node and reach the agent over its stream; a change to the file is
|
||||
noticed and the agent restarts itself.
|
||||
|
||||
### E4. Web settings page (S)
|
||||
The same document, edited on /admin/settings, for whoever has no terminal.
|
||||
|
||||
---
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
|
||||
+126
@@ -16,11 +16,137 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.9.0 | — | Managing people and the tenancy tree | released |
|
||||
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## v0.12.0 — 2026-10-06
|
||||
|
||||
A minor version, because where the configuration lives changes: everything in
|
||||
server.yaml that is not one machine's own is now the cluster's settings, kept
|
||||
in the store, the same on every control plane and in force within seconds of
|
||||
a change. Nothing needs doing by hand to upgrade. The first control plane to
|
||||
start on v0.12.0 brings its server.yaml's cluster keys into the store, with
|
||||
the baseline guard policy when the file had none. From then on, change them
|
||||
with `llmgw ctl config` or the console's Settings, not in server.yaml, whose
|
||||
cluster keys are ignored, with a warning. Upgrade every control plane (`llmgw
|
||||
ctl upgrade` does, the leader last): until the leader runs v0.12.0,
|
||||
`llmgw ctl config` through a standby is refused. Nodes need v0.12.0 to follow
|
||||
the firewall switch.
|
||||
|
||||
### Changed
|
||||
|
||||
- **The cluster's settings live in the store, and every change is in force
|
||||
within seconds.** Everything in server.yaml that is not one machine's own
|
||||
(guardrails, limits and prices, routing, engines, notifications, the
|
||||
mirror's policy, declared models, offline, federation, facility budgets) is
|
||||
a versioned document in the store, the same on every control plane.
|
||||
`llmgw ctl config` shows, sets, edits, lists and rolls it back, and the
|
||||
console's Settings screen edits it. A change is checked, kept with who made
|
||||
it and why, and applied: guardrails live, anything else by every control
|
||||
plane starting again on it, standbys first and the leader last. A change to
|
||||
a machine's own keys in its server.yaml is noticed and applied the same way.
|
||||
No "restart now?" any more, and no second copy to keep in step. The first
|
||||
start against a store brings in the file's cluster keys, with the baseline
|
||||
guard policy when the file had none, so an upgraded cluster gets its
|
||||
guardrails; after that, those keys in a server.yaml are ignored, with a
|
||||
warning.
|
||||
|
||||
### Added
|
||||
|
||||
- **llmgw's firewall can be turned off and on for the whole cluster**:
|
||||
`llmgw ctl config set firewall.enabled false`. Every node's agent takes its
|
||||
table out within seconds, and each control plane's new
|
||||
`llmgw-firewall-sync.path` does the same there; `true` puts the rules back
|
||||
as they were. `firewall_networks` changes reach the nodes the same way.
|
||||
- **`llmgw wipe`** removes llmgw from a machine — services, firewall table,
|
||||
configuration, the authority, secrets, state, users — after listing it and
|
||||
asking for the machine's name; `--models`, `--store` and `--all` take
|
||||
weights, the store's database here, backups and the binary too. Also
|
||||
**Do → Wipe this machine** in the console.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The mirror is given an address the nodes reach**: `server init` and
|
||||
`mirror init` took the machine's first address for the mirror's `--base`,
|
||||
which on a machine with Docker could be 172.17.0.1 — a node then asked its
|
||||
own bridge for every model and was refused. They now take the source of
|
||||
the default route, else the first address not on a container's or a
|
||||
bridge's interface. And `node add` has the node reach the mirror's files
|
||||
before it is added: one that cannot fails the checks, naming the address
|
||||
and what to change.
|
||||
- **A standby that becomes the leader reads the store again**: models added
|
||||
or removed, engine builds chosen and domains held through the leader before
|
||||
it, rather than placing on what it read when it started.
|
||||
- **A node on another network joins**: `node add` set up the node's firewall
|
||||
but left the control plane's to the networks it was installed with, so a node
|
||||
on another subnet was installed and then never heard from (lost). It now lets
|
||||
the node in on the control plane it runs on — its subnet, or its address
|
||||
when a router is between them — and says so; a network already let in
|
||||
changes nothing.
|
||||
|
||||
## v0.11.4 — 2026-10-05
|
||||
|
||||
Two control planes, run in anger on the reference cluster.
|
||||
|
||||
### Added
|
||||
|
||||
- **The console sets up a second control plane**: Do → Add a control plane
|
||||
asks for the machine and a floating address and runs `server add`.
|
||||
- **Who leads, everywhere.** `ctl status` and the console's Overview have a
|
||||
`CONTROL` line: which control plane leads, which stand by, one that does not
|
||||
answer, and whether the floating address is on the leader.
|
||||
- **A standby's console is the leader's.** Its admin API passes `/api/` to the
|
||||
leader over the agent port, with the cluster's certificates both ways, so
|
||||
`sudo llmgw` and `llmgw ctl` show the same cluster on either machine. Control
|
||||
plane certificates are now issued for client use too; `server add` reissues
|
||||
older ones.
|
||||
- **`ctl upgrade` in waves.** Nodes are upgraded one, then two, four… up to
|
||||
`--parallel` (20) at once, so hundreds take minutes; a wave that does not
|
||||
come back stops it. The other control plane is upgraded first, over ssh.
|
||||
- `server add` says when the mirror's `--api-allow` would refuse the new
|
||||
control plane.
|
||||
- **Guardrails in the console**: a view of the policies and what each checks,
|
||||
whether their judge models are serving, where they are attached (attach
|
||||
and detach from it), and what they did in a window. `GET
|
||||
/api/v1/guard/events` answers the same, and `GET /api/v1/guard` now says
|
||||
what each policy does.
|
||||
|
||||
- **A standby's portal is the leader's.** Its client port passes everything
|
||||
but `/health`, `/ready` and `/leader` to the leader, and its admin port the
|
||||
dashboard as well as the API; a proxy left pointing at a standby showed an
|
||||
empty cluster. The leader trusts the control planes' `X-Forwarded-For`.
|
||||
|
||||
- `ctl model serve --tools` says in the catalog that a model makes tool
|
||||
calls, for agents and editors.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Continue's agent did nothing.** The configuration the portal writes told
|
||||
Continue every model uses tools, and nothing of its context: a model whose
|
||||
calls came back as text stalled the agent, and prompts over a CPU
|
||||
replica's 4096 tokens were refused. It now says `tool_use` only for a
|
||||
model whose catalog says tools, and the context and longest answer the
|
||||
gateway allows.
|
||||
- `ctl model serve` gave llama.cpp four slots whatever was asked: a
|
||||
`--context-length 32768 --ctx-pool 32768` model had slots of 8192 and
|
||||
refused what the gateway admitted. Slots are now what the pool holds of
|
||||
the context. The longest answer grows with the context (a quarter, from
|
||||
1024 to 8192) rather than being 1024 for every model.
|
||||
- `ctl upgrade` upgrades the leader last wherever it is run; run on the
|
||||
standby, it upgraded the leader first and the leadership moved twice.
|
||||
- **Failover took 40 seconds; it takes a few.** A stopping control plane held
|
||||
every agent's stream for the thirty seconds of a graceful shutdown; it now
|
||||
gives up its lease first and lets the agents go at once.
|
||||
- A control plane that lost its lease while running kept its agents, leaving
|
||||
the new leader with no nodes. It now lets them go.
|
||||
- An agent's reconnect backoff carried over from one outage to the next, so
|
||||
each failover made the next one slower; it starts again after a session that
|
||||
worked.
|
||||
- The `CONTROL` line named no floating address.
|
||||
|
||||
## v0.11.2 — 2026-10-05
|
||||
|
||||
### Routing by what each vLLM replica's cache holds
|
||||
|
||||
+321
-220
File diff suppressed because it is too large.
Load diff
@@ -71,6 +71,7 @@ message ControlMessage {
|
||||
FabricTestRequest fabric_test = 9;
|
||||
UpgradeAgent upgrade = 10;
|
||||
ControlPlanes control_planes = 11;
|
||||
Firewall firewall = 13;
|
||||
}
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -80,6 +81,21 @@ message ControlMessage {
|
||||
map<string, string> trace = 12;
|
||||
}
|
||||
|
||||
// Firewall is whether llmgw's firewall is kept on the node — its nftables
|
||||
// table, which lets the cluster's networks alone reach the engines' ports and
|
||||
// the node exporter — sent when an agent registers and whenever the
|
||||
// cluster's settings change it (firewall.enabled).
|
||||
message Firewall {
|
||||
bool enabled = 1;
|
||||
// networks are let in besides those the node reaches the control planes
|
||||
// on: firewall_networks, networks or single addresses.
|
||||
repeated string networks = 2;
|
||||
// engine_ports_from and engine_ports_to are the engines' ports
|
||||
// (control.port_range).
|
||||
int32 engine_ports_from = 3;
|
||||
int32 engine_ports_to = 4;
|
||||
}
|
||||
|
||||
// ControlPlanes is every control plane an agent may dial, sent by the leader
|
||||
// when an agent registers and whenever the list changes. The agent keeps it
|
||||
// beside its configuration, so a second control plane added after the node
|
||||
|
||||
+426
-2
@@ -83,6 +83,9 @@
|
||||
{
|
||||
"name": "restart"
|
||||
},
|
||||
{
|
||||
"name": "settings"
|
||||
},
|
||||
{
|
||||
"name": "summary"
|
||||
},
|
||||
@@ -854,6 +857,43 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/guard/events": {
|
||||
"get": {
|
||||
"operationId": "getGuardEvents",
|
||||
"summary": "What the guardrails did: totals and the newest requests",
|
||||
"tags": [
|
||||
"guard"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "since",
|
||||
"in": "query",
|
||||
"description": "How far back: 90m, 24h, 7d; 24h when absent.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/GuardEventsView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/jobs": {
|
||||
"get": {
|
||||
"operationId": "getJobs",
|
||||
@@ -1119,7 +1159,7 @@
|
||||
"delete": {
|
||||
"operationId": "deleteModelsName",
|
||||
"summary": "Remove a model added at runtime",
|
||||
"description": "A model declared in configuration is refused: it would come back on the next restart.",
|
||||
"description": "A model declared in the cluster's settings is refused: it is changed there.",
|
||||
"tags": [
|
||||
"models"
|
||||
],
|
||||
@@ -2926,6 +2966,165 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/settings": {
|
||||
"get": {
|
||||
"operationId": "getSettings",
|
||||
"summary": "The cluster's settings in force, or one earlier version",
|
||||
"tags": [
|
||||
"settings"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "version",
|
||||
"in": "query",
|
||||
"description": "An earlier version; the newest when absent.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SettingsView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
},
|
||||
"put": {
|
||||
"operationId": "putSettings",
|
||||
"summary": "Change the cluster's settings: in force on every control plane within seconds",
|
||||
"tags": [
|
||||
"settings"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SettingsRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SettingsView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/settings/history": {
|
||||
"get": {
|
||||
"operationId": "getSettingsHistory",
|
||||
"summary": "Versions of the cluster's settings, newest first",
|
||||
"tags": [
|
||||
"settings"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "limit",
|
||||
"in": "query",
|
||||
"description": "How many; 20 when absent.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/SettingsView"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/settings/rollback": {
|
||||
"post": {
|
||||
"operationId": "postSettingsRollback",
|
||||
"summary": "Put an earlier version of the cluster's settings back in force",
|
||||
"tags": [
|
||||
"settings"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SettingsRollbackRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SettingsView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/summary": {
|
||||
"get": {
|
||||
"operationId": "getSummary",
|
||||
@@ -3443,6 +3642,45 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"ControlPlane": {
|
||||
"type": "object",
|
||||
"description": "ControlPlane is one control plane.",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"address": {
|
||||
"type": "string"
|
||||
},
|
||||
"state": {
|
||||
"type": "string",
|
||||
"description": "State is \"leading\", \"standing by\" or \"not answering\"."
|
||||
},
|
||||
"this": {
|
||||
"type": "boolean",
|
||||
"description": "This is the control plane that answered the summary."
|
||||
}
|
||||
}
|
||||
},
|
||||
"ControlPlanes": {
|
||||
"type": "object",
|
||||
"description": "ControlPlanes is every control plane of a cluster that has several, as each answered when asked.",
|
||||
"properties": {
|
||||
"members": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/ControlPlane"
|
||||
}
|
||||
},
|
||||
"floating": {
|
||||
"type": "string",
|
||||
"description": "Floating is the address keepalived keeps on the leader (10.0.0.161), and FloatingUp whether it answered as a leader when asked; empty when nothing floats."
|
||||
},
|
||||
"floating_up": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"DomainView": {
|
||||
"type": "object",
|
||||
"description": "DomainView is one domain: its nodes, its power and whether it is held.",
|
||||
@@ -3810,6 +4048,115 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"GuardCountView": {
|
||||
"type": "object",
|
||||
"description": "GuardCountView is how many requests one policy did one thing to.",
|
||||
"properties": {
|
||||
"project": {
|
||||
"type": "string"
|
||||
},
|
||||
"action": {
|
||||
"type": "string"
|
||||
},
|
||||
"policy": {
|
||||
"type": "string"
|
||||
},
|
||||
"category": {
|
||||
"type": "string"
|
||||
},
|
||||
"count": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
}
|
||||
}
|
||||
},
|
||||
"GuardEventView": {
|
||||
"type": "object",
|
||||
"description": "GuardEventView is one request the guardrails did something to.",
|
||||
"properties": {
|
||||
"at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"request_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"project": {
|
||||
"type": "string"
|
||||
},
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "integer"
|
||||
},
|
||||
"action": {
|
||||
"type": "string"
|
||||
},
|
||||
"policy": {
|
||||
"type": "string"
|
||||
},
|
||||
"category": {
|
||||
"type": "string"
|
||||
},
|
||||
"guard_ms": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
}
|
||||
}
|
||||
},
|
||||
"GuardEventsView": {
|
||||
"type": "object",
|
||||
"description": "GuardEventsView is what the guardrails did since a moment: totals, and the newest requests.",
|
||||
"properties": {
|
||||
"since": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"counts": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/GuardCountView"
|
||||
}
|
||||
},
|
||||
"events": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/GuardEventView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"GuardPolicyView": {
|
||||
"type": "object",
|
||||
"description": "GuardPolicyView is what one policy does, in brief.",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"mode": {
|
||||
"type": "string",
|
||||
"description": "Mode is enforce, shadow or off; Fail is open or closed."
|
||||
},
|
||||
"fail": {
|
||||
"type": "string"
|
||||
},
|
||||
"no_store": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"judge": {
|
||||
"type": "string",
|
||||
"description": "Judge is the judge model, when one reads the text."
|
||||
},
|
||||
"checks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Checks are what it looks for, one line each: \"input: pii email → redact\"."
|
||||
}
|
||||
}
|
||||
},
|
||||
"GuardView": {
|
||||
"type": "object",
|
||||
"description": "GuardView is the library and where its policies are attached.",
|
||||
@@ -3831,6 +4178,13 @@
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/GuardAttachmentView"
|
||||
}
|
||||
},
|
||||
"details": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/GuardPolicyView"
|
||||
},
|
||||
"description": "Details says what each policy does, in the order of Policies."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -5158,6 +5512,64 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"SettingsRequest": {
|
||||
"type": "object",
|
||||
"description": "SettingsRequest is a new version of the cluster's settings, made from the version Base, which must still be the newest.",
|
||||
"properties": {
|
||||
"base": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"body": {
|
||||
"type": "string"
|
||||
},
|
||||
"note": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"SettingsRollbackRequest": {
|
||||
"type": "object",
|
||||
"description": "SettingsRollbackRequest puts an earlier version's settings back in force, as a new version.",
|
||||
"properties": {
|
||||
"version": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"note": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"SettingsView": {
|
||||
"type": "object",
|
||||
"description": "SettingsView is one version of the cluster's settings.",
|
||||
"properties": {
|
||||
"version": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"body": {
|
||||
"type": "string"
|
||||
},
|
||||
"note": {
|
||||
"type": "string"
|
||||
},
|
||||
"at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"by": {
|
||||
"type": "string"
|
||||
},
|
||||
"local_keys": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ShardView": {
|
||||
"type": "object",
|
||||
"description": "ShardView is one shard of a replica.",
|
||||
@@ -5224,6 +5636,10 @@
|
||||
"type": "object",
|
||||
"description": "Summary is the whole fleet in one answer.\n\nA per-node listing is the right tool for four nodes and the wrong one for two hundred: the questions that matter at that size — how full am I, what is broken, what is holding cards it is not using — are aggregates, and reading them off a long table is a chore that gets skipped. Everything here is derived from the same state the tables show, so it can never disagree with them; it is a different question asked of the same facts.",
|
||||
"properties": {
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Version is the release the answering control plane runs: not the binary on its disk, which a manual install changes underneath it."
|
||||
},
|
||||
"nodes": {
|
||||
"$ref": "#/components/schemas/NodeTotals"
|
||||
},
|
||||
@@ -5276,6 +5692,10 @@
|
||||
},
|
||||
"store_error": {
|
||||
"type": "string"
|
||||
},
|
||||
"control_planes": {
|
||||
"$ref": "#/components/schemas/ControlPlanes",
|
||||
"description": "ControlPlanes is the cluster's control planes when there is more than one: which leads, which stand by, and the floating address in front."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -6805,11 +7225,15 @@
|
||||
},
|
||||
"ctx_pool": {
|
||||
"type": "integer",
|
||||
"description": "CtxPool is llama.cpp's --ctx-size; ContextLength what one request may use. They default to 16384 and 4096, what four parallel slots divide."
|
||||
"description": "CtxPool is llama.cpp's --ctx-size; ContextLength what one request may use. They default to 16384 and 4096, what four parallel slots divide; llama.cpp is given as many slots as the pool holds requests."
|
||||
},
|
||||
"context_length": {
|
||||
"type": "integer"
|
||||
},
|
||||
"tools": {
|
||||
"type": "boolean",
|
||||
"description": "Tools says the model makes tool calls the engine returns as calls: it is written into the catalog, where clients and editors read it. Check a running replica with llmgw ctl tool-check before saying so."
|
||||
},
|
||||
"cpu_cores": {
|
||||
"type": "integer",
|
||||
"description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on."
|
||||
|
||||
@@ -184,6 +184,15 @@ asks for, send the results back. Whether a model calls tools well is the
|
||||
model's business, not the gateway's; ask your operator which of the served
|
||||
models are set up for it.
|
||||
|
||||
**Editors' agents need such a model.** Continue's Agent mode, Cline and
|
||||
Codex act only through tool calls: with a model whose calls come back as
|
||||
text — `<tool_call>{...}` or a JSON block in the answer, `tool_calls` empty —
|
||||
they chat but never read a file or run a command. Qwen2.5-Coder is one;
|
||||
Qwen3-Coder is not. The configuration the portal writes for Continue says
|
||||
`tool_use` only for a model whose catalog entry says tools, and gives
|
||||
Continue the model's context and longest answer, so it never sends a request
|
||||
the gateway refuses.
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
|
||||
+17
-8
@@ -270,14 +270,20 @@ page's text.
|
||||
|
||||
## Settings
|
||||
|
||||
**server.yaml** (on a node, **agent.yaml**) is every key of the file, as a
|
||||
tree of menus built from the configuration reference: the value set, or the
|
||||
one that applies when nothing is, its type, and what it is for. Enter changes
|
||||
a value or opens a block; on a list or map of blocks, Enter opens an entry and
|
||||
**D** removes one. **U** puts a key back to its default. **S** saves: the file
|
||||
is checked the way the service reads it before anything is written, and its
|
||||
comments are kept. The service reads it when it starts, so the console offers
|
||||
to restart it. Another control plane keeps its own copy.
|
||||
**server.yaml** (on a node, **agent.yaml**) is every key, as a tree of menus
|
||||
built from the configuration reference: the value set, or the one that
|
||||
applies when nothing is, its type, and what it is for. Enter changes a value
|
||||
or opens a block; on a list or map of blocks, Enter opens an entry and **D**
|
||||
removes one. **U** puts a key back to its default. **S** saves.
|
||||
|
||||
On a control plane, the cluster's keys are read from and saved to the
|
||||
cluster's settings in the store — the same on every control plane, in force
|
||||
on all of them within seconds (`llmgw ctl config` does the same from a
|
||||
shell) — and this machine's own keys (listeners, certificates, the store's
|
||||
address) to its server.yaml, which the server notices and applies by itself.
|
||||
Either is checked before anything is kept, and the file's comments survive.
|
||||
On a node, agent.yaml is read when the agent starts, so the console offers
|
||||
to restart it.
|
||||
|
||||
## Do
|
||||
|
||||
@@ -313,6 +319,9 @@ to restart it. Another control plane keeps its own copy.
|
||||
- **Restart all replicas**, one at a time.
|
||||
- **Back up now** — the store and `/etc/llmgw` into one archive
|
||||
(`sudo llmgw server backup`).
|
||||
- **Wipe this machine** — removes llmgw's services, configuration, state and
|
||||
users from this machine, after listing them and asking for its name
|
||||
(`sudo llmgw wipe`; on a node too).
|
||||
- **Leave**.
|
||||
|
||||
## The node's console
|
||||
|
||||
+18
-3
@@ -5,9 +5,13 @@ policy can check, what it can do, and how one is attached to a team.
|
||||
|
||||
`llmgw server init` writes a `baseline` policy that applies to every
|
||||
request, with Llama Guard 3 1B as its judge on a CPU; see
|
||||
[install.md](install.md#2-the-control-plane) for what it checks. Without a
|
||||
`guard:` section in `server.yaml` nothing is checked and nothing costs
|
||||
anything. With one, every request is judged once it is authorised and
|
||||
[install.md](install.md#2-the-control-plane) for what it checks. The
|
||||
policies are part of the cluster's settings, kept in the store: `llmgw ctl
|
||||
config edit` (or the console's Settings) changes them, and the change is in
|
||||
force on every control plane within seconds, without a restart. A cluster
|
||||
whose settings were brought in from a server.yaml without a `guard:` section
|
||||
gets the `baseline` policy, without a judge. With no policies, nothing is
|
||||
checked and nothing costs anything. With one, every request is judged once it is authorised and
|
||||
within its limits, with its stored conversation spliced in, and before it is
|
||||
sent to a model; and the answer is judged before the client sees it.
|
||||
|
||||
@@ -209,3 +213,14 @@ Each request's usage row has the action, policy, category and the time the
|
||||
checks took. The access log has the action, stage and policy. Neither has
|
||||
the text, a matched value, or the placeholder map: the map exists only in
|
||||
the gateway's memory while the request is served.
|
||||
|
||||
## In the console
|
||||
|
||||
`sudo llmgw` has a **Guardrails** view: each policy with its mode, what it
|
||||
does when it cannot judge, and what it checks; whether each judge model it
|
||||
leans on has replicas ready — a policy that fails closed refuses everything
|
||||
while its judge has none; where policies are attached, with A to attach one
|
||||
and Enter on a row to detach it; and what they did in the last day (W for an
|
||||
hour or a week), totals first and the newest requests after, each with the
|
||||
request id `llmgw ctl request` reads. The same is at `GET /api/v1/guard` and
|
||||
`GET /api/v1/guard/events?since=24h`, and on the portal's Guardrails page.
|
||||
+147
-28
@@ -241,8 +241,38 @@ password, change it, and issue yourself a key from the profile page.
|
||||
|
||||
### The configuration
|
||||
|
||||
`/etc/llmgw/server.yaml` is yours from here on. Three things in it are worth
|
||||
knowing before you edit it:
|
||||
There are two halves, and each change to either is in force within seconds,
|
||||
on every machine it concerns, without anyone restarting anything.
|
||||
|
||||
**The cluster's settings** — guardrails, limits and prices, routing, the
|
||||
reconciler, engines, notifications, the mirror's policy, models declared by
|
||||
hand, offline, federation, facility budgets, backups' recipients — are kept in
|
||||
the store, so every control plane runs on the same ones. Change them with
|
||||
|
||||
```
|
||||
llmgw ctl config # what is in force, and its version
|
||||
llmgw ctl config set router.retries 3 --note "flaky upstream"
|
||||
llmgw ctl config edit # the whole document in $EDITOR
|
||||
llmgw ctl config history # who changed what, when, and why
|
||||
llmgw ctl config rollback 4 # back to version 4, as a new version
|
||||
```
|
||||
|
||||
or on the console's **Settings** screen. A change is checked before it is
|
||||
kept — one that would not load is refused — and recorded in the audit log.
|
||||
Guardrails change live; anything else has every control plane start again on
|
||||
the new settings, the standbys first and the leader half a minute later, so
|
||||
one of them is always serving. Engines keep running through it.
|
||||
|
||||
The first control plane to start against a store brings in its server.yaml's
|
||||
cluster keys — with a `baseline` guard policy, if the file had none — and
|
||||
from then on those keys in any server.yaml are not read (the server says so
|
||||
when it starts).
|
||||
|
||||
**This machine's own keys** stay in `/etc/llmgw/server.yaml`: `listen`,
|
||||
`admin_listen`, `metrics_listen`, `agent_listen`, `agent_tls`, `auth.store`,
|
||||
`secrets`, `tracing`, `monitoring`, `mirror.api` and `mirror.tls`, the backup
|
||||
directory and key. The server notices the file change and starts again on it
|
||||
by itself. Three things are worth knowing before you edit it:
|
||||
|
||||
- `admin_listen` is on loopback on purpose, and the server refuses any other
|
||||
address unless `admin_remote: true`. Its API under `/api` adds models and
|
||||
@@ -1915,15 +1945,53 @@ one trusted machine only; see [development.md](development.md).
|
||||
|
||||
## Two control planes
|
||||
|
||||
llmgw handles the hard half itself: only one places work, and agents find
|
||||
whichever is leading. What it does not do is move client traffic, which
|
||||
happens in front of both.
|
||||
Once the store is replicated (`server init --patroni`, above), one command on
|
||||
the control plane makes another machine a second one — or **Do → Add a
|
||||
control plane** in the console, which asks the same two questions:
|
||||
|
||||
[deploy/ha/](../deploy/ha/) has the pattern and a keepalived example. The one
|
||||
thing to get right is the health check: point it at `/ready`, not `/health`.
|
||||
`/health` is 200 on a standby, which is alive and holds no endpoints at all,
|
||||
so a load balancer using it would send half the traffic to something that
|
||||
refuses every request.
|
||||
```
|
||||
sudo --preserve-env=SSH_AUTH_SOCK llmgw server add ubuntu@10.0.0.150 --vip 10.0.0.161/24
|
||||
```
|
||||
|
||||
Over ssh, as `node add` works, it gives the machine this llmgw, a certificate
|
||||
from the cluster's authority, this control plane's `server.yaml` and secrets,
|
||||
and lets it into the store; it starts there as a standby. Every node is told
|
||||
of it within half a minute and fails over to it without being touched.
|
||||
`--vip` keeps a floating address on whichever control plane leads, with
|
||||
keepalived on both: that is the address to give clients, or a proxy in front.
|
||||
Without it, put a load balancer in front that checks `/ready` — not
|
||||
`/health`, which is 200 on a standby that would refuse every request.
|
||||
|
||||
- **What a failover costs.** Measured on the reference cluster, stopping the
|
||||
leader: the standby led within five seconds, every node had moved to it
|
||||
and every replica was adopted — none restarted, the engines never stopped
|
||||
serving — and the floating address answered again with it.
|
||||
- **Who leads.** `llmgw ctl status` and the console's Overview have a
|
||||
`CONTROL` line: which leads, which stand by, one that does not answer, and
|
||||
whether the floating address is on the leader. A standby with nothing to
|
||||
take over from is the thing to see before it is needed.
|
||||
- **Either machine, the leader's cluster.** A standby passes its client port
|
||||
— the portal, its admin pages, the API — to the leader's, keeping only
|
||||
`/health`, `/ready` and `/leader` to describe itself, so a proxy pointed at
|
||||
the wrong machine still shows the cluster; the leader trusts a control
|
||||
plane's `X-Forwarded-For`, so clients keep their own address. Its admin
|
||||
API passes everything but `/debug/` and `/metrics` to the leader, over the port control planes already reach each
|
||||
other on and with the cluster's certificates both ways, so `sudo llmgw` and
|
||||
`llmgw ctl` show the same cluster on both — the leader's — and a change made
|
||||
on the standby is made by the leader. A node's certificate is refused
|
||||
there. Control plane certificates from before v0.11.3 cannot present
|
||||
themselves as clients: `server add` reissues them, and the running server
|
||||
picks the new one up when restarted.
|
||||
- **The mirror** answers only the addresses it was given with `--api-allow`.
|
||||
`server add` checks, and says what to add to the mirror's unit if it
|
||||
refuses the new control plane — which would otherwise find out the first
|
||||
time it leads and fetches a model.
|
||||
- **Limits are per process.** Rate limits are counted in memory, and clients
|
||||
reach the leader through the floating address, so a failover starts each
|
||||
key's minute afresh.
|
||||
|
||||
[deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
|
||||
their own load balancer in front.
|
||||
|
||||
## The portal's language
|
||||
|
||||
@@ -2063,14 +2131,33 @@ loaded at boot by `llmgw-firewall.service`):
|
||||
|
||||
"The cluster's networks" are, on each machine, the network of the interface
|
||||
it reaches the control planes (or the other store members) on — or just
|
||||
their address, when the route goes through a gateway. Anything else that
|
||||
must reach those ports, a Prometheus on another subnet or nodes behind a
|
||||
router, goes in `server.yaml` before the machines are set up:
|
||||
their address, when the route goes through a gateway. `node add` opens
|
||||
the control plane's to each node it adds when the node is on a network not
|
||||
already let in: its subnet when the control plane is on it, its address alone
|
||||
when a router is between them. Anything else that must reach those ports, a
|
||||
Prometheus on another subnet, a second control plane's or a separate mirror's
|
||||
for nodes behind a router, goes in `server.yaml`:
|
||||
|
||||
```yaml
|
||||
firewall_networks: [10.0.5.0/24, 192.168.1.20]
|
||||
```
|
||||
|
||||
**Turning it off and on.** Whether llmgw keeps this firewall is one of the
|
||||
cluster's settings:
|
||||
|
||||
```
|
||||
llmgw ctl config set firewall.enabled false # every machine's table taken out
|
||||
llmgw ctl config set firewall.enabled true # and put back as it was
|
||||
```
|
||||
|
||||
It is in force within seconds: each node's agent takes its table out or puts
|
||||
it back, and each control plane's `llmgw-firewall-sync.path` (installed by
|
||||
`server init` and `server add`; `sudo llmgw firewall sync --install` on one
|
||||
set up before) does the same there. The rules files stay, so turning it back
|
||||
on restores them. Nodes added while it is off get none. A separate mirror
|
||||
machine and the store's members keep theirs for now. `firewall_networks`
|
||||
changes reach the nodes the same way.
|
||||
|
||||
Only those ports are touched. The table's policy is to accept, so whatever
|
||||
else the machine serves — and whatever firewall it already has — is left as
|
||||
it was; the client API (`:8080`) is not closed, being meant to be reached.
|
||||
@@ -2207,6 +2294,31 @@ A cluster set up online becomes offline the same way: `bundle import` on the
|
||||
mirror and control plane, then the `offline:` block in `server.yaml` and a
|
||||
restart.
|
||||
|
||||
## Removing llmgw from a machine
|
||||
|
||||
```
|
||||
sudo llmgw wipe # services, configuration, certificates, state, users
|
||||
sudo llmgw wipe --models # and model weights, the mirror's files, engines
|
||||
sudo llmgw wipe --store # and the store's database here (Patroni member, etcd, or local PostgreSQL)
|
||||
sudo llmgw wipe --all # all of that, backups and the binary
|
||||
```
|
||||
|
||||
It lists everything it will remove and asks for the machine's name (`--yes`
|
||||
does not ask), then stops llmgw's services and the engines, takes the
|
||||
firewall table out, and deletes `/etc/llmgw` — the authority and secrets
|
||||
with it — `/var/lib/llmgw`, the agent's state, the sudo grant, the
|
||||
authority from the system's trust, keepalived's configuration when llmgw
|
||||
wrote it, and the `llmgw` and `llmgw-engine` users. Packages it installed
|
||||
(PostgreSQL, Patroni, etcd, nftables, the node exporter) stay. Run it again
|
||||
to finish what failed. A node wiped is then forgotten on the control plane
|
||||
with `llmgw node remove <name>`. The console has it too: **Do → Wipe this
|
||||
machine**.
|
||||
|
||||
This is how a machine with an older install is set up again from nothing:
|
||||
wipe it, then install as above. Wiping a control plane without `--store`
|
||||
leaves the store with the cluster's accounts, keys and settings in it, and a
|
||||
new control plane against that store takes them up.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
The control plane runs in Kubernetes from the Helm chart in
|
||||
@@ -2215,27 +2327,34 @@ joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
|
||||
|
||||
## Upgrading
|
||||
|
||||
On the control plane:
|
||||
On a control plane:
|
||||
|
||||
```
|
||||
sudo llmgw ctl upgrade v0.10.4
|
||||
sudo --preserve-env=SSH_AUTH_SOCK llmgw ctl upgrade v0.10.4
|
||||
```
|
||||
|
||||
It upgrades this machine first — fetching the release for its platform,
|
||||
checking it against the release's checksums, keeping the old binary as
|
||||
`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
|
||||
for every node to reconnect — and then each node in turn. A node's agent is told
|
||||
the version over the stream it already holds, fetches and checks the release
|
||||
itself, swaps it in and restarts; no ssh into the nodes is needed. Release
|
||||
candidates (`v0.10.4-rc.2`) work the same way.
|
||||
With two control planes it upgrades the standby first and the leader last,
|
||||
whichever this machine is — the other over ssh as the user who ran sudo
|
||||
(`--ssh-user`), with `llmgw upgrade` there, and this machine —
|
||||
fetching the release for its platform, checking it against the release's
|
||||
checksums, keeping the old binary as `/usr/local/bin/llmgw.bak-pre-<version>`,
|
||||
restarting `llmgw-server` and waiting for every node to reconnect. The
|
||||
leader's restart hands over once, to a standby already on the release.
|
||||
|
||||
Then the nodes, in waves: one first, then two, four, and so on up to
|
||||
`--parallel` (20) at once. A node's agent is told the version over the stream
|
||||
it already holds, fetches and checks the release itself, swaps it in and
|
||||
restarts; no ssh into the nodes is needed. A fleet of hundreds takes minutes,
|
||||
not hours. Release candidates (`v0.10.4-rc.2`) work the same way.
|
||||
|
||||
A node counts as done when it is back on the new version, ready, with as many
|
||||
ready replicas as before. One that is not within `--timeout` (three minutes)
|
||||
stops the upgrade there, so a bad release reaches one node rather than all of
|
||||
them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
|
||||
beside the new one. Nodes already on the version are skipped, so running the
|
||||
command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
|
||||
and `--skip-control-plane` leaves this machine alone.
|
||||
ready replicas as before. A wave not back within `--timeout` (three minutes)
|
||||
stops the upgrade there — the first wave is one node, so a bad release reaches
|
||||
one node rather than all of them; `llmgw ctl logs <node>` says why, and the node
|
||||
keeps its previous binary beside the new one. Nodes already on the version are
|
||||
skipped, so running the command again resumes where it stopped. `--nodes a,b`
|
||||
limits it to some nodes, `--skip-control-plane` leaves the control planes alone,
|
||||
and `--other-control-planes=false` this machine's partner.
|
||||
|
||||
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
|
||||
keep running when the agent restarts, and the agent reports them back on its
|
||||
|
||||
@@ -98,6 +98,7 @@ rather than repeated.
|
||||
| `other_watts` | `float64` | [FacilityBudget](#facilitybudget) |
|
||||
| `name` | `string` | [Federation](#federation) |
|
||||
| `interval` | `time.Duration` | [Federation](#federation) |
|
||||
| `enabled` | `bool` | [FirewallSettings](#firewallsettings) |
|
||||
| `format` | `string` | [GuardModel](#guardmodel) |
|
||||
| `categories` | `[]string` | [GuardModel](#guardmodel) |
|
||||
| `model` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
@@ -287,6 +288,7 @@ rather than repeated.
|
||||
| `guard` | `guard.Config` | [Server](#server) |
|
||||
| `secrets` | `Secrets` | [Server](#server) |
|
||||
| `notify` | `notify.Config` | [Server](#server) |
|
||||
| `firewall` | `FirewallSettings` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
| `backup_copy_to` | `[]string` | [Server](#server) |
|
||||
@@ -963,6 +965,19 @@ Interval is how often each peer is asked whether it is ready, timed,
|
||||
and what it offers. A peer that does not answer is taken out of
|
||||
routing until it does. Zero is 10s.
|
||||
|
||||
## FirewallSettings
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
|
||||
Enabled keeps llmgw's internal ports — engines, node exporter, the
|
||||
agents' stream, the mirror, the store — to the cluster's networks on
|
||||
every machine. False removes llmgw's nftables table everywhere, for a
|
||||
site whose own firewall does that job; the rules are kept, and true
|
||||
puts them back. Changed in the cluster's settings, it is in force on
|
||||
every node and control plane within seconds.
|
||||
|
||||
## GuardModel
|
||||
|
||||
### `format`
|
||||
@@ -2441,6 +2456,14 @@ Prometheus and Alertmanager in between. Empty sends nothing.
|
||||
|
||||
Takes a [notify.Config](#notifyconfig) block.
|
||||
|
||||
### `firewall`
|
||||
|
||||
`FirewallSettings`
|
||||
|
||||
Firewall turns llmgw's firewall on every machine on or off.
|
||||
|
||||
Takes a [FirewallSettings](#firewallsettings) block.
|
||||
|
||||
### `firewall_networks`
|
||||
|
||||
`[]string`
|
||||
|
||||
+26
-6
@@ -21,6 +21,7 @@ import (
|
||||
"slices"
|
||||
"strconv"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
@@ -35,6 +36,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
@@ -42,11 +44,14 @@ import (
|
||||
|
||||
// Agent runs on a GPU node.
|
||||
type Agent struct {
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
sup *supervisor.Supervisor
|
||||
reg *engines.Registry
|
||||
http connectHTTPClient
|
||||
cfg Config
|
||||
// answered says the current session heard from its control plane, which
|
||||
// starts the reconnect backoff again when it ends.
|
||||
answered atomic.Bool
|
||||
log *slog.Logger
|
||||
sup *supervisor.Supervisor
|
||||
reg *engines.Registry
|
||||
http connectHTTPClient
|
||||
|
||||
// engineOnce probes the configured engines once before registering.
|
||||
//
|
||||
@@ -83,7 +88,11 @@ type Agent struct {
|
||||
// serverList is every control plane to dial: the configured ones, and
|
||||
// any the leader has named since. Read through servers().
|
||||
serverList []string
|
||||
inv discovery.Inventory
|
||||
// fwMu keeps one firewall change at a time; fwRunner runs its scripts
|
||||
// (nil: this machine's shell).
|
||||
fwMu sync.Mutex
|
||||
fwRunner firewall.Runner
|
||||
inv discovery.Inventory
|
||||
// shardReplica remembers which replica a shard belongs to, so a status
|
||||
// report can name both. The control plane keys on the replica.
|
||||
shardReplica map[string]string
|
||||
@@ -215,6 +224,14 @@ func (a *Agent) Run(ctx context.Context) error {
|
||||
// never answers — from here they are the same thing, and trying
|
||||
// somewhere else is the answer to all three.
|
||||
next++
|
||||
// A session that was answered was a connection that worked, however
|
||||
// it ended: the backoff starts again rather than carrying on from
|
||||
// the last outage. Without this every failover the agent lived
|
||||
// through made the next one slower — the third waited half a minute
|
||||
// on a control plane that had been ready for twenty seconds.
|
||||
if a.answered.Swap(false) {
|
||||
attempt = 0
|
||||
}
|
||||
attempt++
|
||||
delay := a.reconnectDelay(attempt)
|
||||
// No wait when there is somewhere else to try: the delay exists to
|
||||
@@ -568,6 +585,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
// why it is not registering is one an operator cannot fix.
|
||||
return err
|
||||
}
|
||||
a.answered.Store(true)
|
||||
a.handle(ctx, msg, out)
|
||||
}
|
||||
}
|
||||
@@ -841,6 +859,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
|
||||
go a.upgrade(ctx, m.Upgrade, out)
|
||||
case *llmgwv1.ControlMessage_ControlPlanes:
|
||||
a.setControlPlanes(m.ControlPlanes)
|
||||
case *llmgwv1.ControlMessage_Firewall:
|
||||
go a.applyFirewall(m.Firewall)
|
||||
case *llmgwv1.ControlMessage_Logs:
|
||||
reply := a.logsReply(m.Logs)
|
||||
// Off the receive loop: a full send queue must not stop commands being
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -105,3 +106,49 @@ func TestALeaderStillAcceptsWithNoLeadershipConfigured(t *testing.T) {
|
||||
}
|
||||
t.Fatal("a service with no leadership configured refused an agent")
|
||||
}
|
||||
|
||||
// flipping is a leader that can stop leading partway through a test.
|
||||
type flipping struct{ leading atomic.Bool }
|
||||
|
||||
func (f *flipping) IsLeader() bool { return f.leading.Load() }
|
||||
|
||||
func TestAnAgentLeavesAControlPlaneThatStopsLeading(t *testing.T) {
|
||||
// Leadership lost while the process lives on: the store was slow and the
|
||||
// lease lapsed, or another control plane took it. Agents that stayed on
|
||||
// the old leader would leave the new one with no nodes, so the old one
|
||||
// has to let them go rather than wait for them to leave.
|
||||
was := &flipping{}
|
||||
was.leading.Store(true)
|
||||
wasInv, wasAddr := startPlain(t, was)
|
||||
nowInv, nowAddr := startPlain(t, standby(false))
|
||||
|
||||
ag := runAgent(t, Config{
|
||||
NodeID: "n1", MgmtAddr: "127.0.0.1",
|
||||
Servers: []string{wasAddr, nowAddr},
|
||||
})
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
go ag.Run(ctx)
|
||||
|
||||
waitFor(t, 5*time.Second, "the agent never reached the first leader", func() bool {
|
||||
_, ok := wasInv.Get("n1")
|
||||
return ok
|
||||
})
|
||||
was.leading.Store(false)
|
||||
waitFor(t, 8*time.Second, "the agent stayed on a control plane that stopped leading", func() bool {
|
||||
_, ok := nowInv.Get("n1")
|
||||
return ok
|
||||
})
|
||||
}
|
||||
|
||||
func waitFor(t *testing.T, d time.Duration, msg string, ok func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(d)
|
||||
for time.Now().Before(deadline) {
|
||||
if ok() {
|
||||
return
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
t.Fatal(msg)
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
)
|
||||
|
||||
// applyFirewall keeps llmgw's firewall on this node as the cluster's
|
||||
// settings say: the engines' ports and the node exporter's for the networks
|
||||
// the node reaches its control planes on, and those named, or no table at
|
||||
// all. Off the receive loop, one at a time.
|
||||
func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
a.fwMu.Lock()
|
||||
defer a.fwMu.Unlock()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
defer cancel()
|
||||
r := a.fwRunner
|
||||
if r == nil {
|
||||
r = firewall.Local{}
|
||||
}
|
||||
if !m.GetEnabled() {
|
||||
had, err := firewall.Disable(ctx, r)
|
||||
switch {
|
||||
case err != nil:
|
||||
a.log.Warn("could not take llmgw's firewall off this node", "err", err)
|
||||
case had:
|
||||
a.log.Info("llmgw's firewall is off on this node, as the cluster's settings say")
|
||||
}
|
||||
return
|
||||
}
|
||||
a.mu.Lock()
|
||||
servers := append([]string(nil), a.serverList...)
|
||||
a.mu.Unlock()
|
||||
nets, err := firewall.Networks(ctx, r, controlPlaneIPs(ctx, servers))
|
||||
if err != nil {
|
||||
a.log.Warn("could not tell which networks reach the control planes; the firewall is left as it is", "err", err)
|
||||
return
|
||||
}
|
||||
for _, n := range m.GetNetworks() {
|
||||
if p, err := netip.ParsePrefix(n); err == nil {
|
||||
nets = append(nets, p)
|
||||
} else if ad, err := netip.ParseAddr(n); err == nil {
|
||||
nets = append(nets, netip.PrefixFrom(ad, ad.BitLen()))
|
||||
}
|
||||
}
|
||||
from, to := int(m.GetEnginePortsFrom()), int(m.GetEnginePortsTo())
|
||||
ports := []firewall.Port{firewall.NodeExporter}
|
||||
if from > 0 && to >= from {
|
||||
ports = append(ports, firewall.Engines(from, to))
|
||||
}
|
||||
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
|
||||
switch {
|
||||
case err != nil:
|
||||
a.log.Warn("could not put llmgw's firewall on this node", "err", err)
|
||||
case changed:
|
||||
a.log.Info("llmgw's firewall is on this node", "networks", len(nets))
|
||||
}
|
||||
}
|
||||
|
||||
// controlPlaneIPs are the addresses of the control planes an agent dials.
|
||||
func controlPlaneIPs(ctx context.Context, servers []string) []string {
|
||||
var out []string
|
||||
for _, s := range servers {
|
||||
host := s
|
||||
if u, err := url.Parse(s); err == nil && u.Host != "" {
|
||||
host = u.Hostname()
|
||||
}
|
||||
host = strings.Trim(host, "[]")
|
||||
if a, err := netip.ParseAddr(host); err == nil {
|
||||
out = append(out, a.String())
|
||||
continue
|
||||
}
|
||||
ips, err := net.DefaultResolver.LookupNetIP(ctx, "ip", host)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, ip := range ips {
|
||||
out = append(out, ip.Unmap().String())
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// fakeShell answers the firewall package's scripts and records what was
|
||||
// installed or removed.
|
||||
type fakeShell struct {
|
||||
present bool
|
||||
installed string
|
||||
deleted bool
|
||||
}
|
||||
|
||||
func (f *fakeShell) Run(_ context.Context, script string, stdin []byte) ([]byte, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(script, "ip -o route get 10.0.0.140"):
|
||||
return []byte("10.0.0.140 dev eth0 src 10.0.0.141 uid 0\n2: eth0 inet 10.0.0.141/24 brd 10.0.0.255 scope global eth0\n"), nil
|
||||
case strings.HasPrefix(script, "sudo -n /usr/sbin/nft list table"):
|
||||
if f.present {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, errors.New("no table")
|
||||
case strings.Contains(script, "nft delete table") && !strings.Contains(script, "cat > "):
|
||||
f.deleted, f.present = true, false
|
||||
return nil, nil
|
||||
case strings.HasPrefix(script, "sudo cat "):
|
||||
return nil, nil
|
||||
default:
|
||||
f.installed, f.present = string(stdin), true
|
||||
return []byte("changed\n"), nil
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
|
||||
sh := &fakeShell{}
|
||||
a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"}}
|
||||
|
||||
a.applyFirewall(&llmgwv1.Firewall{Enabled: true, Networks: []string{"192.168.5.0/24", "10.9.9.9"},
|
||||
EnginePortsFrom: 20000, EnginePortsTo: 20999})
|
||||
for _, want := range []string{"10.0.0.0/24", "192.168.5.0/24", "10.9.9.9/32", "20000-20999", "9100"} {
|
||||
if !strings.Contains(sh.installed, want) {
|
||||
t.Errorf("the rules lack %s:\n%s", want, sh.installed)
|
||||
}
|
||||
}
|
||||
|
||||
a.applyFirewall(&llmgwv1.Firewall{Enabled: false})
|
||||
if !sh.deleted {
|
||||
t.Error("turned off, the table stayed")
|
||||
}
|
||||
}
|
||||
@@ -54,7 +54,7 @@ func newBackupCmd() *cobra.Command {
|
||||
"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
|
||||
"BackupTooOld fires when it is more than a day old.",
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
cfg, err := config.Load(configPath)
|
||||
cfg, err := loadEffectiveConfig(configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -323,7 +323,7 @@ func loadServerConfig(path string) (config.Server, bool) {
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
return config.Server{}, false
|
||||
}
|
||||
cfg, err := config.Load(path)
|
||||
cfg, err := loadEffectiveConfig(path)
|
||||
if err != nil {
|
||||
return config.Server{}, false
|
||||
}
|
||||
|
||||
@@ -0,0 +1,363 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// settingsPoll is how often a control plane asks whether the cluster's
|
||||
// settings, or its own server.yaml, changed.
|
||||
const settingsPoll = 5 * time.Second
|
||||
|
||||
// clusterSettings is what a control plane runs on, and where it came from:
|
||||
// its server.yaml for its own keys, the store's settings for the rest.
|
||||
type clusterSettings struct {
|
||||
cfg config.Server
|
||||
eff map[string]any // the settings, as written, cfg was made from
|
||||
cluster map[string]any // the store's document, nil without a store
|
||||
version int64
|
||||
fileHash [32]byte
|
||||
}
|
||||
|
||||
// loadClusterSettings reads the configuration a control plane starts on.
|
||||
// Without a store it is the file. With one, the store's settings take the
|
||||
// place of the file's cluster keys; a store with none yet is given the
|
||||
// file's, so the first control plane to start against it brings them in.
|
||||
func loadClusterSettings(ctx context.Context, path string, log *slog.Logger) (clusterSettings, error) {
|
||||
var s clusterSettings
|
||||
b, file, err := readSettingsFile(path)
|
||||
if err != nil {
|
||||
return s, err
|
||||
}
|
||||
s.fileHash = sha256.Sum256(b)
|
||||
fileCfg, err := config.FromMap(file)
|
||||
if err != nil {
|
||||
return s, fileError(path, err)
|
||||
}
|
||||
if fileCfg.Auth.Store == "" {
|
||||
s.cfg, s.eff = fileCfg, file
|
||||
return s, nil
|
||||
}
|
||||
st, err := storepkg.Open(ctx, fileCfg.Auth.Store)
|
||||
if err != nil {
|
||||
return s, fmt.Errorf("the store, for the cluster's settings: %w", err)
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
doc, err := st.ClusterSettings(ctx)
|
||||
if err != nil {
|
||||
return s, fmt.Errorf("reading the cluster's settings: %w", err)
|
||||
}
|
||||
if doc.Version == 0 {
|
||||
if doc, err = importSettings(ctx, st, path, file, log); err != nil {
|
||||
return s, err
|
||||
}
|
||||
} else if ignored := config.Ignored(file); len(ignored) > 0 {
|
||||
log.Warn("server.yaml sets keys the cluster's settings now hold; they are not read "+
|
||||
"(llmgw ctl config shows what is in force)", "keys", strings.Join(ignored, ", "))
|
||||
}
|
||||
// The newest version that this machine can run on. One that cannot — a
|
||||
// path only another control plane has — must not keep this one down,
|
||||
// where nothing could be used to put it right.
|
||||
versions := []storepkg.SettingsVersion{doc}
|
||||
if older, err := st.ClusterSettingsHistory(ctx, 20); err == nil && len(older) > 1 {
|
||||
versions = append(versions, older[1:]...)
|
||||
}
|
||||
var firstErr error
|
||||
for _, v := range versions {
|
||||
cluster, err := config.Parse([]byte(v.Body))
|
||||
if err == nil {
|
||||
eff := config.Layer(file, cluster)
|
||||
var cfg config.Server
|
||||
if cfg, err = config.FromMap(eff); err == nil {
|
||||
if v.Version != doc.Version {
|
||||
log.Error("the cluster's newest settings do not load on this machine; running on an older version",
|
||||
"newest", doc.Version, "running", v.Version, "err", firstErr)
|
||||
}
|
||||
s.cfg, s.eff, s.cluster, s.version = cfg, eff, cluster, doc.Version
|
||||
return s, nil
|
||||
}
|
||||
}
|
||||
if firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
return s, fmt.Errorf("the cluster's settings (version %d) do not load on this machine: %w", doc.Version, firstErr)
|
||||
}
|
||||
|
||||
// importSettings records a server.yaml's cluster keys as the cluster's first
|
||||
// settings, with the baseline guard policy when the file has none: a cluster
|
||||
// installed before guardrails gets them on its first start after an upgrade.
|
||||
func importSettings(ctx context.Context, st *storepkg.DB, path string, file map[string]any,
|
||||
log *slog.Logger,
|
||||
) (storepkg.SettingsVersion, error) {
|
||||
_, cluster := config.Split(file)
|
||||
note := "imported from " + path
|
||||
if _, ok := cluster["guard"]; !ok {
|
||||
baseline, err := config.Parse([]byte(guardSection(false)))
|
||||
if err == nil {
|
||||
cluster["guard"] = baseline["guard"]
|
||||
note += ", with the baseline guard policy it did not have"
|
||||
}
|
||||
}
|
||||
body, err := config.Marshal(cluster)
|
||||
if err != nil {
|
||||
return storepkg.SettingsVersion{}, err
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
_, err = st.PutClusterSettings(ctx, 0, string(body), host, note)
|
||||
switch {
|
||||
case errors.Is(err, storepkg.ErrSettingsChanged):
|
||||
// Another control plane brought its own in first: those are the
|
||||
// cluster's.
|
||||
case err != nil:
|
||||
return storepkg.SettingsVersion{}, fmt.Errorf("recording the cluster's settings: %w", err)
|
||||
default:
|
||||
log.Info("the cluster's settings are now kept in the store; server.yaml keeps this machine's own",
|
||||
"from", path)
|
||||
}
|
||||
return st.ClusterSettings(ctx)
|
||||
}
|
||||
|
||||
func readSettingsFile(path string) ([]byte, map[string]any, error) {
|
||||
if path == "" {
|
||||
return nil, map[string]any{}, nil
|
||||
}
|
||||
b, err := os.ReadFile(path) //nolint:gosec // the configuration
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("config file: %w", err)
|
||||
}
|
||||
m, err := config.Parse(b)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("parsing %s: %w", path, err)
|
||||
}
|
||||
return b, m, nil
|
||||
}
|
||||
|
||||
func fileError(path string, err error) error {
|
||||
if path == "" {
|
||||
return err
|
||||
}
|
||||
return fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
|
||||
// settingsWatch follows the cluster's settings and this machine's
|
||||
// server.yaml, and puts every change in force.
|
||||
type settingsWatch struct {
|
||||
path string
|
||||
st *storepkg.DB // nil: the file only
|
||||
cur clusterSettings
|
||||
// live applies a change without a restart, reporting whether it could.
|
||||
live func(cfg config.Server, changed []string) bool
|
||||
// restart has the process start again on the new settings.
|
||||
restart func(reason string)
|
||||
log *slog.Logger
|
||||
}
|
||||
|
||||
func (w *settingsWatch) run(ctx context.Context) {
|
||||
t := time.NewTicker(settingsPoll)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
w.check(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// check reads what changed, if anything, and applies it.
|
||||
func (w *settingsWatch) check(ctx context.Context) {
|
||||
version := w.cur.version
|
||||
if w.st != nil {
|
||||
v, err := w.st.ClusterSettingsVersion(ctx)
|
||||
if err != nil {
|
||||
return // asked again in a few seconds
|
||||
}
|
||||
version = v
|
||||
}
|
||||
b, file, err := readSettingsFile(w.path)
|
||||
if err != nil {
|
||||
w.log.Error("server.yaml cannot be read; running on what it said before", "err", err)
|
||||
return
|
||||
}
|
||||
hash := sha256.Sum256(b)
|
||||
if version == w.cur.version && hash == w.cur.fileHash {
|
||||
return
|
||||
}
|
||||
cluster := w.cur.cluster
|
||||
eff := file
|
||||
if w.st != nil {
|
||||
if version != w.cur.version {
|
||||
doc, err := w.st.ClusterSettings(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if cluster, err = config.Parse([]byte(doc.Body)); err != nil {
|
||||
w.log.Error("the cluster's settings do not parse; running on what was in force",
|
||||
"version", doc.Version, "err", err)
|
||||
w.cur.version, w.cur.fileHash = version, hash
|
||||
return
|
||||
}
|
||||
}
|
||||
eff = config.Layer(file, cluster)
|
||||
}
|
||||
cfg, err := config.FromMap(eff)
|
||||
if err != nil {
|
||||
w.log.Error("the changed settings do not load on this machine; running on what was in force",
|
||||
"version", version, "err", err)
|
||||
w.cur.version, w.cur.fileHash = version, hash
|
||||
return
|
||||
}
|
||||
changed := config.Changed(w.cur.eff, eff)
|
||||
w.cur = clusterSettings{cfg: cfg, eff: eff, cluster: cluster, version: version, fileHash: hash}
|
||||
if len(changed) == 0 {
|
||||
return
|
||||
}
|
||||
if w.live != nil && w.live(cfg, changed) {
|
||||
w.log.Info("settings changed and in force", "version", version, "changed", strings.Join(changed, ", "))
|
||||
return
|
||||
}
|
||||
w.restart(fmt.Sprintf("settings version %d changed %s", version, strings.Join(changed, ", ")))
|
||||
}
|
||||
|
||||
// liveSettings are the keys a control plane puts in force without starting
|
||||
// again: what it reads at every use, or not at all.
|
||||
var liveSettings = []string{
|
||||
"guard",
|
||||
// Sent to the nodes and this machine's firewall unit as they change.
|
||||
"firewall", "firewall_networks",
|
||||
// Read by server init, node add and backups, not by the server.
|
||||
"backup_recipients", "backup_copy_to",
|
||||
}
|
||||
|
||||
// onlyLive is whether every changed key is one of liveSettings.
|
||||
func onlyLive(changed []string) bool {
|
||||
for _, c := range changed {
|
||||
top, _, _ := strings.Cut(c, ".")
|
||||
if !slices.Contains(liveSettings, top) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// checkClusterSettings refuses a settings document that would not load,
|
||||
// laid over the server.yaml at path, or that sets one machine's keys.
|
||||
func checkClusterSettings(path string, body []byte) error {
|
||||
cluster, err := config.Parse(body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := config.CheckCluster(cluster); err != nil {
|
||||
return err
|
||||
}
|
||||
_, file, err := readSettingsFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = config.FromMap(config.Layer(file, cluster))
|
||||
return err
|
||||
}
|
||||
|
||||
// loadEffectiveConfig is what the control plane on this machine runs on:
|
||||
// its server.yaml with the cluster's settings from the store laid over it,
|
||||
// for the commands beside it that read cluster keys — init, node add,
|
||||
// backups. The file alone when the store cannot be reached or holds none.
|
||||
func loadEffectiveConfig(path string) (config.Server, error) {
|
||||
_, file, err := readSettingsFile(path)
|
||||
if err != nil {
|
||||
return config.Default(), err
|
||||
}
|
||||
fileCfg, err := config.FromMap(file)
|
||||
if err != nil {
|
||||
return fileCfg, fileError(path, err)
|
||||
}
|
||||
cluster, ok := storedSettings(fileCfg)
|
||||
if !ok {
|
||||
return fileCfg, nil
|
||||
}
|
||||
cfg, err := config.FromMap(config.Layer(file, cluster))
|
||||
if err != nil {
|
||||
return fileCfg, nil //nolint:nilerr // the file still says what this machine is
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// storedSettings are the cluster's settings, when the store a configuration
|
||||
// names can be reached and holds some.
|
||||
func storedSettings(cfg config.Server) (map[string]any, bool) {
|
||||
if cfg.Auth.Store == "" {
|
||||
return nil, false
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
st, err := storepkg.Open(ctx, cfg.Auth.Store)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
defer st.Close()
|
||||
doc, err := st.ClusterSettings(ctx)
|
||||
if err != nil || doc.Version == 0 {
|
||||
return nil, false
|
||||
}
|
||||
cluster, err := config.Parse([]byte(doc.Body))
|
||||
return cluster, err == nil
|
||||
}
|
||||
|
||||
// changeStoredSettings changes the cluster's settings from a command run on
|
||||
// a control plane (init), when the store holds them: changed is false when
|
||||
// it does not, and the file is then where the change belongs.
|
||||
func changeStoredSettings(path, note string, change func(doc *settings.Doc) error) (changed bool, err error) {
|
||||
_, file, err := readSettingsFile(path)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
fileCfg, err := config.FromMap(file)
|
||||
if err != nil || fileCfg.Auth.Store == "" {
|
||||
return false, nil //nolint:nilerr // no store to keep them in
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
st, err := storepkg.Open(ctx, fileCfg.Auth.Store)
|
||||
if err != nil {
|
||||
return false, nil //nolint:nilerr // not set up yet: the file it is
|
||||
}
|
||||
defer st.Close()
|
||||
cur, err := st.ClusterSettings(ctx)
|
||||
if err != nil || cur.Version == 0 {
|
||||
return false, err
|
||||
}
|
||||
doc, err := settings.FromBytes("the cluster's settings", []byte(cur.Body))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if err := change(doc); err != nil {
|
||||
return false, err
|
||||
}
|
||||
body, err := doc.Bytes()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if err := checkClusterSettings(path, body); err != nil {
|
||||
return false, err
|
||||
}
|
||||
who := "llmgw"
|
||||
if u := os.Getenv("SUDO_USER"); u != "" {
|
||||
who = u
|
||||
}
|
||||
_, err = st.PutClusterSettings(ctx, cur.Version, string(body), who, note)
|
||||
return err == nil, err
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func settingsFixture(t *testing.T, extra string) (path, store string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
store = filepath.Join(dir, "llmgw.db")
|
||||
path = filepath.Join(dir, "server.yaml")
|
||||
body := "listen: \":8080\"\nagent_insecure: true\nauth:\n enabled: true\n store: " + store +
|
||||
"\nrouter:\n retries: 2\n" + extra
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path, store
|
||||
}
|
||||
|
||||
var quiet = slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||
|
||||
func TestTheFirstStartBringsTheFilesSettingsInWithGuardrails(t *testing.T) {
|
||||
path, store := settingsFixture(t, "")
|
||||
s, err := loadClusterSettings(t.Context(), path, quiet)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if s.version != 1 || s.cfg.Router.Retries != 2 {
|
||||
t.Errorf("version %d retries %d", s.version, s.cfg.Router.Retries)
|
||||
}
|
||||
if len(s.cfg.Guard.Policies) == 0 || s.cfg.Guard.Policies["baseline"].Judge != nil {
|
||||
t.Errorf("a file without guard got %+v, want the baseline without a judge", s.cfg.Guard)
|
||||
}
|
||||
st, _ := storepkg.Open(t.Context(), store)
|
||||
defer st.Close()
|
||||
doc, _ := st.ClusterSettings(t.Context())
|
||||
if strings.Contains(doc.Body, "listen") || strings.Contains(doc.Body, store) || !strings.Contains(doc.Body, "retries: 2") {
|
||||
t.Errorf("the store holds a machine's keys, or lost the cluster's:\n%s", doc.Body)
|
||||
}
|
||||
|
||||
// From then on the store's settings are the cluster's, whatever a
|
||||
// file says.
|
||||
_ = os.WriteFile(path, []byte(strings.Replace(mustRead(t, path), "retries: 2", "retries: 9", 1)), 0o600)
|
||||
again, err := loadClusterSettings(t.Context(), path, quiet)
|
||||
if err != nil || again.version != 1 || again.cfg.Router.Retries != 2 {
|
||||
t.Errorf("second start: version %d retries %d err %v", again.version, again.cfg.Router.Retries, err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRead(t *testing.T, p string) string {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func TestEveryChangeIsAppliedLiveOrByStartingAgain(t *testing.T) {
|
||||
path, store := settingsFixture(t, "")
|
||||
s, err := loadClusterSettings(t.Context(), path, quiet)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, _ := storepkg.Open(t.Context(), store)
|
||||
defer st.Close()
|
||||
var lived [][]string
|
||||
var restarts []string
|
||||
w := &settingsWatch{path: path, st: st, cur: s, log: quiet,
|
||||
live: func(_ config.Server, changed []string) bool {
|
||||
lived = append(lived, changed)
|
||||
return onlyLive(changed)
|
||||
},
|
||||
restart: func(why string) { restarts = append(restarts, why) },
|
||||
}
|
||||
ctx := context.Background()
|
||||
put := func(body string) {
|
||||
t.Helper()
|
||||
v, _ := st.ClusterSettingsVersion(ctx)
|
||||
if _, err := st.PutClusterSettings(ctx, v, body, "test", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
cur, _ := st.ClusterSettings(ctx)
|
||||
|
||||
w.check(ctx) // nothing changed
|
||||
if len(lived)+len(restarts) != 0 {
|
||||
t.Fatalf("acted on no change: %v %v", lived, restarts)
|
||||
}
|
||||
|
||||
put(strings.Replace(cur.Body, "injection: flag", "injection: block", 1))
|
||||
w.check(ctx)
|
||||
if len(restarts) != 0 || len(lived) != 1 || lived[0][0] != "guard" {
|
||||
t.Fatalf("a guard change: lived %v restarts %v", lived, restarts)
|
||||
}
|
||||
|
||||
cur, _ = st.ClusterSettings(ctx)
|
||||
put(strings.Replace(cur.Body, "retries: 2", "retries: 4", 1))
|
||||
w.check(ctx)
|
||||
if len(restarts) != 1 || !strings.Contains(restarts[0], "router") || w.cur.cfg.Router.Retries != 4 {
|
||||
t.Fatalf("a router change: restarts %v retries %d", restarts, w.cur.cfg.Router.Retries)
|
||||
}
|
||||
|
||||
// A machine's own key, in its file.
|
||||
_ = os.WriteFile(path, []byte(strings.Replace(mustRead(t, path), `":8080"`, `":8090"`, 1)), 0o600)
|
||||
w.check(ctx)
|
||||
if len(restarts) != 2 || !strings.Contains(restarts[1], "listen") {
|
||||
t.Fatalf("a listener change: %v", restarts)
|
||||
}
|
||||
|
||||
// Settings that do not load are not put in force.
|
||||
cur, _ = st.ClusterSettings(ctx)
|
||||
put(cur.Body + "control:\n timezone: Mars/Olympus\n")
|
||||
w.check(ctx)
|
||||
if len(restarts) != 2 || w.cur.cfg.Control.Timezone != "" {
|
||||
t.Errorf("bad settings were applied: %v %q", restarts, w.cur.cfg.Control.Timezone)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClusterDocumentCannotSetAMachinesKeys(t *testing.T) {
|
||||
path, _ := settingsFixture(t, "")
|
||||
if err := checkClusterSettings(path, []byte("listen: \":1\"\n")); err == nil {
|
||||
t.Error("listen was accepted")
|
||||
}
|
||||
if err := checkClusterSettings(path, []byte("control:\n timezone: Mars/Olympus\n")); err == nil {
|
||||
t.Error("a timezone that is not one was accepted")
|
||||
}
|
||||
if err := checkClusterSettings(path, []byte("router:\n retries: 3\n")); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// floatingPath is where `server add --vip` records the floating address.
|
||||
const floatingPath = "/etc/llmgw/floating"
|
||||
|
||||
// controlPlaneProbe answers the summary's question of which control plane
|
||||
// leads: it asks each one's /leader, on the client port every one of them
|
||||
// listens on, since they share a server.yaml.
|
||||
//
|
||||
// Asked, rather than read from the lease, because the lease names whoever
|
||||
// held it last and says nothing about a standby that has stopped — and a
|
||||
// standby that is not there is the thing worth seeing before it is needed.
|
||||
type controlPlaneProbe struct {
|
||||
st *store.DB
|
||||
self string
|
||||
// floating is the file the floating address is recorded in.
|
||||
floating string
|
||||
client *http.Client
|
||||
// leaderURL is where a host is asked; tests point it elsewhere.
|
||||
leaderURL func(host string) string
|
||||
|
||||
mu sync.Mutex
|
||||
at time.Time
|
||||
last adminapi.ControlPlanes
|
||||
}
|
||||
|
||||
func newControlPlaneProbe(st *store.DB, listen string) *controlPlaneProbe {
|
||||
_, port, _ := net.SplitHostPort(listen)
|
||||
if port == "" {
|
||||
port = "8080"
|
||||
}
|
||||
self, _ := os.Hostname()
|
||||
return &controlPlaneProbe{st: st, self: self,
|
||||
leaderURL: func(host string) string { return "http://" + net.JoinHostPort(host, port) + "/leader" },
|
||||
floating: floatingPath,
|
||||
client: &http.Client{Timeout: time.Second}}
|
||||
}
|
||||
|
||||
// Status is the answer, kept for a few seconds: the console asks every three,
|
||||
// and each asking is a request to every control plane.
|
||||
func (p *controlPlaneProbe) Status() adminapi.ControlPlanes {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if time.Since(p.at) < 2*time.Second {
|
||||
return p.last
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
cps, err := p.st.ControlPlanes(ctx)
|
||||
if err != nil || len(cps) < 2 {
|
||||
// One control plane is not a set of them; nothing to say.
|
||||
p.at, p.last = time.Now(), adminapi.ControlPlanes{}
|
||||
return p.last
|
||||
}
|
||||
// Read each time: server add writes it after this process may have
|
||||
// started.
|
||||
out := adminapi.ControlPlanes{Members: make([]adminapi.ControlPlane, len(cps)), Floating: floatingAddress(p.floating)}
|
||||
var wg sync.WaitGroup
|
||||
for i, cp := range cps {
|
||||
host := cp.URL
|
||||
if u, err := url.Parse(cp.URL); err == nil && u.Hostname() != "" {
|
||||
host = u.Hostname()
|
||||
}
|
||||
out.Members[i] = adminapi.ControlPlane{Name: cp.Name, Address: host, This: cp.Name == p.self}
|
||||
wg.Go(func() { out.Members[i].State = p.ask(ctx, host) })
|
||||
}
|
||||
if out.Floating != "" {
|
||||
wg.Go(func() { out.FloatingUp = p.ask(ctx, out.Floating) == "leading" })
|
||||
}
|
||||
wg.Wait()
|
||||
p.at, p.last = time.Now(), out
|
||||
return out
|
||||
}
|
||||
|
||||
// ask is what one address's /leader says.
|
||||
func (p *controlPlaneProbe) ask(ctx context.Context, host string) string {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, p.leaderURL(host), nil)
|
||||
if err != nil {
|
||||
return "not answering"
|
||||
}
|
||||
resp, err := p.client.Do(req)
|
||||
if err != nil {
|
||||
return "not answering"
|
||||
}
|
||||
_ = resp.Body.Close()
|
||||
switch resp.StatusCode {
|
||||
case http.StatusOK:
|
||||
return "leading"
|
||||
case http.StatusServiceUnavailable:
|
||||
return "standing by"
|
||||
}
|
||||
return "not answering"
|
||||
}
|
||||
|
||||
// floatingAddress is the address in the file server add keeps beside the
|
||||
// keepalived configuration, without its prefix; "" when nothing floats.
|
||||
// keepalived's own file holds its password and is root's alone, and this
|
||||
// process is not root.
|
||||
func floatingAddress(path string) string {
|
||||
b, err := os.ReadFile(path) //nolint:gosec // a fixed path of our own
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
addr, _, _ := strings.Cut(strings.TrimSpace(string(b)), "/")
|
||||
if net.ParseIP(addr) == nil {
|
||||
return ""
|
||||
}
|
||||
return addr
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestStatusSaysWhichControlPlaneLeadsAndWhetherTheOthersStandBy(t *testing.T) {
|
||||
render := func(s adminapi.Summary) string {
|
||||
cmd := &cobra.Command{}
|
||||
var b bytes.Buffer
|
||||
cmd.SetOut(&b)
|
||||
printSummary(cmd, s)
|
||||
return b.String()
|
||||
}
|
||||
out := render(adminapi.Summary{ControlPlanes: adminapi.ControlPlanes{
|
||||
Members: []adminapi.ControlPlane{
|
||||
{Name: "llmgw-ctl", State: "leading", This: true},
|
||||
{Name: "llmgw-ctl02", State: "standing by"},
|
||||
},
|
||||
Floating: "10.0.0.161", FloatingUp: true,
|
||||
}})
|
||||
if !strings.Contains(out, "CONTROL llmgw-ctl leads (this one), llmgw-ctl02 standing by · 10.0.0.161 on the leader") {
|
||||
t.Errorf("status:\n%s", out)
|
||||
}
|
||||
out = render(adminapi.Summary{ControlPlanes: adminapi.ControlPlanes{
|
||||
Members: []adminapi.ControlPlane{
|
||||
{Name: "llmgw-ctl", State: "not answering"},
|
||||
{Name: "llmgw-ctl02", State: "standing by", This: true},
|
||||
},
|
||||
Floating: "10.0.0.161",
|
||||
}})
|
||||
for _, want := range []string{"NO LEADER", "llmgw-ctl not answering", "10.0.0.161 NOT ANSWERING"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("%q missing:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
if out := render(adminapi.Summary{}); strings.Contains(out, "CONTROL") {
|
||||
t.Error("a single control plane got a control line")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheProbeAsksEachControlPlaneWhetherItLeads(t *testing.T) {
|
||||
leading := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
|
||||
defer leading.Close()
|
||||
standby := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
}))
|
||||
defer standby.Close()
|
||||
gone := httptest.NewServer(http.NotFoundHandler())
|
||||
gone.Close()
|
||||
|
||||
st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
for _, cp := range []store.ControlPlane{
|
||||
{URL: "https://10.0.0.1:7443", Name: "one"},
|
||||
{URL: "https://10.0.0.2:7443", Name: "two"},
|
||||
{URL: "https://10.0.0.3:7443", Name: "three"},
|
||||
} {
|
||||
if err := st.PutControlPlane(t.Context(), cp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
at := map[string]string{"10.0.0.1": leading.URL, "10.0.0.2": standby.URL, "10.0.0.3": gone.URL, "10.0.0.161": leading.URL}
|
||||
p := newControlPlaneProbe(st, ":8080")
|
||||
floating := filepath.Join(t.TempDir(), "floating")
|
||||
if err := os.WriteFile(floating, []byte("10.0.0.161/24\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p.self, p.floating = "two", floating
|
||||
p.leaderURL = func(host string) string { return at[host] + "/leader" }
|
||||
|
||||
s := p.Status()
|
||||
got := map[string]adminapi.ControlPlane{}
|
||||
for _, m := range s.Members {
|
||||
got[m.Name] = m
|
||||
}
|
||||
if got["one"].State != "leading" || got["two"].State != "standing by" || got["three"].State != "not answering" {
|
||||
t.Errorf("states: %+v", got)
|
||||
}
|
||||
if !got["two"].This || got["one"].This || got["one"].Address != "10.0.0.1" {
|
||||
t.Errorf("which and where: %+v", got)
|
||||
}
|
||||
if !s.FloatingUp {
|
||||
t.Error("a floating address on the leader read as down")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOneControlPlaneIsNotASetOfThem(t *testing.T) {
|
||||
st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
if err := st.PutControlPlane(t.Context(), store.ControlPlane{URL: "https://10.0.0.1:7443", Name: "one"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if s := newControlPlaneProbe(st, ":8080").Status(); len(s.Members) != 0 {
|
||||
t.Errorf("one control plane reported as a set: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheFloatingAddressIsReadFromWhatServerAddWrote(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, "floating")
|
||||
if err := os.WriteFile(file, []byte("10.0.0.161/24\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := floatingAddress(file); got != "10.0.0.161" {
|
||||
t.Errorf("read %q", got)
|
||||
}
|
||||
if err := os.WriteFile(file, []byte("not an address\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := floatingAddress(file); got != "" {
|
||||
t.Errorf("nonsense read as %q", got)
|
||||
}
|
||||
if got := floatingAddress(filepath.Join(dir, "none")); got != "" {
|
||||
t.Errorf("no file read as %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheProbeReadsTheFloatingAddressFromItsFile(t *testing.T) {
|
||||
// The file, not what it held when the process started: server add
|
||||
// writes it after restarting the control plane it adds.
|
||||
if p := newControlPlaneProbe(nil, ":8080"); p.floating != floatingPath {
|
||||
t.Errorf("the probe reads %q, not %s", p.floating, floatingPath)
|
||||
}
|
||||
}
|
||||
@@ -39,6 +39,14 @@ func newCtlClient(addr string, tokenFiles ...string) *ctlClient {
|
||||
}
|
||||
}
|
||||
|
||||
// patient is the same client waiting up to d for an answer, for a call the
|
||||
// server itself may spend minutes on.
|
||||
func (c *ctlClient) patient(d time.Duration) *ctlClient {
|
||||
cp := *c
|
||||
cp.http = &http.Client{Timeout: d}
|
||||
return &cp
|
||||
}
|
||||
|
||||
func (c *ctlClient) get(ctx context.Context, path string, out any) error {
|
||||
return c.do(ctx, http.MethodGet, path, nil, out)
|
||||
}
|
||||
@@ -165,6 +173,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newCapacityCmd(client, &asJSON),
|
||||
newQuotaCmd(client, &asJSON),
|
||||
newGuardCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
newAuditCmd(client, &asJSON),
|
||||
newNotifyCmd(client, &asJSON),
|
||||
newUsageCmd(client, &asJSON),
|
||||
|
||||
@@ -0,0 +1,377 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
func newConfigCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var version int64
|
||||
c := &cobra.Command{
|
||||
Use: "config",
|
||||
Short: "The cluster's settings: show, change, and go back",
|
||||
Long: "Everything in server.yaml that is not one machine's own — guardrails,\n" +
|
||||
"limits, pricing, routing, engines, notifications, the mirror's policy —\n" +
|
||||
"is the cluster's, kept in the store and the same on every control plane.\n" +
|
||||
"A change is checked, kept as a new version, and in force on every control\n" +
|
||||
"plane within seconds: live where it can be (guard), otherwise each control\n" +
|
||||
"plane starts again on it, standbys first and the leader last.\n\n" +
|
||||
"With no subcommand, prints the settings in force.\n\n" +
|
||||
"A machine's own keys (" + strings.Join(config.LocalKeys(), ", ") + ")\n" +
|
||||
"stay in its /etc/llmgw/server.yaml; a change there is noticed and applied\n" +
|
||||
"on that machine the same way.",
|
||||
Example: " llmgw ctl config\n" +
|
||||
" llmgw ctl config get router.retries\n" +
|
||||
" llmgw ctl config set router.retries 3 --note 'flaky upstream'\n" +
|
||||
" llmgw ctl config set firewall_networks '[10.20.0.0/16]'\n" +
|
||||
" llmgw ctl config edit\n" +
|
||||
" llmgw ctl config history\n" +
|
||||
" llmgw ctl config rollback 4",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
path := "/api/v1/settings"
|
||||
if version > 0 {
|
||||
path += "?version=" + strconv.FormatInt(version, 10)
|
||||
}
|
||||
var v adminapi.SettingsView
|
||||
if err := client().get(cmd.Context(), path, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(v)
|
||||
}
|
||||
fmt.Fprintf(cmd.ErrOrStderr(), "# version %d, %s by %s%s\n", v.Version, v.At.Local().Format(time.DateTime),
|
||||
orUnknown(v.By), noteSuffix(v.Note))
|
||||
fmt.Fprint(cmd.OutOrStdout(), v.Body)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().Int64Var(&version, "version", 0, "an earlier version")
|
||||
c.AddCommand(newConfigGetCmd(client), newConfigSetCmd(client), newConfigUnsetCmd(client),
|
||||
newConfigEditCmd(client), newConfigHistoryCmd(client, asJSON), newConfigRollbackCmd(client))
|
||||
return c
|
||||
}
|
||||
|
||||
func orUnknown(s string) string {
|
||||
if s == "" {
|
||||
return "unknown"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func noteSuffix(n string) string {
|
||||
if n == "" {
|
||||
return ""
|
||||
}
|
||||
return ": " + n
|
||||
}
|
||||
|
||||
// keyPath reads a dotted key, refusing one machine's own.
|
||||
func keyPath(key string) ([]string, error) {
|
||||
path := strings.Split(key, ".")
|
||||
if config.IsLocal(path) {
|
||||
return nil, fmt.Errorf("%s is each machine's own: change it in that machine's /etc/llmgw/server.yaml, "+
|
||||
"where a change is applied by itself too", key)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
func newConfigGetCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "get KEY",
|
||||
Short: "One key of the cluster's settings, as YAML",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
path, err := keyPath(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var v adminapi.SettingsView
|
||||
if err := client().get(cmd.Context(), "/api/v1/settings", &v); err != nil {
|
||||
return err
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(v.Body))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
n := doc.Node(path)
|
||||
if n == nil {
|
||||
return fmt.Errorf("%s is not set in the cluster's settings; its default applies (docs/reference.md)", args[0])
|
||||
}
|
||||
sub, err := settings.FromBytes("value", nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := sub.SetNode([]string{"v"}, n); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := sub.Bytes()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out := strings.TrimPrefix(string(b), "v:")
|
||||
fmt.Fprintln(cmd.OutOrStdout(), strings.TrimSpace(dedent(out)))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// dedent takes off the two spaces a block nested under one key carries.
|
||||
func dedent(s string) string {
|
||||
lines := strings.Split(s, "\n")
|
||||
for i, l := range lines {
|
||||
lines[i] = strings.TrimPrefix(l, " ")
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// changeSettings reads the newest settings, changes them, and puts the
|
||||
// result in force, saying what that does.
|
||||
func changeSettings(cmd *cobra.Command, client func() *ctlClient, note string,
|
||||
change func(doc *settings.Doc) error,
|
||||
) error {
|
||||
ctx := cmd.Context()
|
||||
var cur adminapi.SettingsView
|
||||
if err := client().get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := change(doc); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := doc.Bytes()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return putSettings(cmd, client, cur, b, note)
|
||||
}
|
||||
|
||||
func putSettings(cmd *cobra.Command, client func() *ctlClient, cur adminapi.SettingsView, body []byte, note string) error {
|
||||
before, _ := config.Parse([]byte(cur.Body))
|
||||
after, err := config.Parse(body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
changed := config.Changed(before, after)
|
||||
if len(changed) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "nothing changed")
|
||||
return nil
|
||||
}
|
||||
req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
|
||||
var v adminapi.SettingsView
|
||||
if err := client().put(cmd.Context(), "/api/v1/settings", req, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "version %d: %s\n", v.Version, strings.Join(changed, ", "))
|
||||
if onlyLive(changed) {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "in force on every control plane within seconds")
|
||||
} else {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "every control plane starts again on it within seconds, standbys first, "+
|
||||
"the leader half a minute later; engines keep serving")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func newConfigSetCmd(client func() *ctlClient) *cobra.Command {
|
||||
var note string
|
||||
c := &cobra.Command{
|
||||
Use: "set KEY VALUE",
|
||||
Short: "Change one key of the cluster's settings; VALUE is YAML",
|
||||
Example: " llmgw ctl config set router.retries 3\n" +
|
||||
" llmgw ctl config set guard.default '[baseline, finance]'\n" +
|
||||
" llmgw ctl config set notify.webhooks '[{url: https://hooks.example/llmgw}]'",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
path, err := keyPath(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
return doc.SetYAML(path, args[1])
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the version")
|
||||
return c
|
||||
}
|
||||
|
||||
func newConfigUnsetCmd(client func() *ctlClient) *cobra.Command {
|
||||
var note string
|
||||
c := &cobra.Command{
|
||||
Use: "unset KEY",
|
||||
Short: "Remove one key from the cluster's settings, so its default applies",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
path, err := keyPath(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if doc.Node(path) == nil {
|
||||
return fmt.Errorf("%s is not set", args[0])
|
||||
}
|
||||
doc.Unset(path)
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the version")
|
||||
return c
|
||||
}
|
||||
|
||||
func newConfigEditCmd(client func() *ctlClient) *cobra.Command {
|
||||
var note string
|
||||
c := &cobra.Command{
|
||||
Use: "edit",
|
||||
Short: "Edit the cluster's settings in $EDITOR",
|
||||
Long: "Opens the settings in force in $EDITOR (vi when unset). Saved, they are\n" +
|
||||
"checked and kept as a new version; a document that does not load is\n" +
|
||||
"opened again with the reason, until it does or is left unchanged.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var cur adminapi.SettingsView
|
||||
if err := client().get(cmd.Context(), "/api/v1/settings", &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
dir, err := os.MkdirTemp("", "llmgw-settings")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
file := filepath.Join(dir, "settings.yaml")
|
||||
body := []byte(cur.Body)
|
||||
problem := ""
|
||||
for {
|
||||
text := body
|
||||
if problem != "" {
|
||||
text = append([]byte("# NOT SAVED: "+strings.ReplaceAll(problem, "\n", " ")+"\n"), stripProblem(body)...)
|
||||
}
|
||||
if err := os.WriteFile(file, text, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := runEditor(file, cmd.InOrStdin(), cmd.OutOrStdout(), cmd.ErrOrStderr()); err != nil {
|
||||
return err
|
||||
}
|
||||
edited, err := os.ReadFile(file) //nolint:gosec // written above
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
edited = stripProblem(edited)
|
||||
if bytes.Equal(bytes.TrimSpace(edited), bytes.TrimSpace([]byte(cur.Body))) {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "nothing changed")
|
||||
return nil
|
||||
}
|
||||
err = putSettings(cmd, client, cur, edited, note)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if strings.Contains(err.Error(), "changed since they were read") {
|
||||
// Somebody else's change came first: opening the editor
|
||||
// again on the old base would only be refused again.
|
||||
return fmt.Errorf("%w; your edit is not saved", err)
|
||||
}
|
||||
body, problem = edited, err.Error()
|
||||
}
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the version")
|
||||
return c
|
||||
}
|
||||
|
||||
func stripProblem(b []byte) []byte {
|
||||
if bytes.HasPrefix(b, []byte("# NOT SAVED: ")) {
|
||||
if i := bytes.IndexByte(b, '\n'); i >= 0 {
|
||||
return b[i+1:]
|
||||
}
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func runEditor(file string, in io.Reader, out, errOut io.Writer) error {
|
||||
editor := os.Getenv("VISUAL")
|
||||
if editor == "" {
|
||||
editor = os.Getenv("EDITOR")
|
||||
}
|
||||
if editor == "" {
|
||||
editor = "vi"
|
||||
}
|
||||
parts := strings.Fields(editor)
|
||||
c := exec.Command(parts[0], append(parts[1:], file)...) //nolint:gosec // the operator's own editor
|
||||
c.Stdin, c.Stdout, c.Stderr = in, out, errOut
|
||||
if err := c.Run(); err != nil {
|
||||
return fmt.Errorf("%s: %w", editor, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func newConfigHistoryCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var limit int
|
||||
c := &cobra.Command{
|
||||
Use: "history",
|
||||
Short: "Versions of the cluster's settings, newest first",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var list []adminapi.SettingsView
|
||||
if err := client().get(cmd.Context(), "/api/v1/settings/history?limit="+strconv.Itoa(limit), &list); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(list)
|
||||
}
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
fmt.Fprintln(w, "VERSION\tWHEN\tBY\tNOTE")
|
||||
for _, v := range list {
|
||||
fmt.Fprintf(w, "%d\t%s\t%s\t%s\n", v.Version, v.At.Local().Format(time.DateTime), orUnknown(v.By), v.Note)
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
c.Flags().IntVar(&limit, "limit", 20, "how many")
|
||||
return c
|
||||
}
|
||||
|
||||
func newConfigRollbackCmd(client func() *ctlClient) *cobra.Command {
|
||||
var note string
|
||||
c := &cobra.Command{
|
||||
Use: "rollback VERSION",
|
||||
Short: "Put an earlier version of the cluster's settings back in force, as a new version",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
n, err := strconv.ParseInt(args[0], 10, 64)
|
||||
if err != nil || n < 1 {
|
||||
return errors.New("VERSION is a number from 'llmgw ctl config history'")
|
||||
}
|
||||
var cur, old adminapi.SettingsView
|
||||
if err := client().get(cmd.Context(), "/api/v1/settings", &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := client().get(cmd.Context(), "/api/v1/settings?version="+args[0], &old); err != nil {
|
||||
return err
|
||||
}
|
||||
if note == "" {
|
||||
note = "back to version " + args[0]
|
||||
}
|
||||
return putSettings(cmd, client, cur, []byte(old.Body), note)
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the version")
|
||||
return c
|
||||
}
|
||||
+216
-64
@@ -16,6 +16,7 @@ import (
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
@@ -27,19 +28,25 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
|
||||
releases string
|
||||
perNode time.Duration
|
||||
serverSvc string
|
||||
parallel int
|
||||
sshUser string
|
||||
othersCP bool
|
||||
)
|
||||
c := &cobra.Command{
|
||||
Use: "upgrade VERSION",
|
||||
Short: "Upgrade this control plane, then every node, one at a time",
|
||||
Short: "Upgrade the control planes, then every node, in waves",
|
||||
Long: "Moves the cluster to a release: v0.10.4, or a candidate like v0.10.4-rc.2.\n\n" +
|
||||
"This machine first — run it with sudo on the control plane, since it\n" +
|
||||
"replaces /usr/local/bin/llmgw and restarts llmgw-server. Then each node, in\n" +
|
||||
"turn: its agent fetches the release for its own platform, checks it\n" +
|
||||
"against the release's checksums, swaps it in and restarts. Engines keep\n" +
|
||||
"serving through all of it and are adopted again; no ssh into the nodes is\n" +
|
||||
"needed.\n\n" +
|
||||
"The other control planes first, over ssh as --ssh-user (the one sudo was\n" +
|
||||
"run by), then this machine — run it with sudo on a control plane, since it\n" +
|
||||
"replaces /usr/local/bin/llmgw and restarts llmgw-server. A leader restarted\n" +
|
||||
"last hands over to a standby already on the release.\n\n" +
|
||||
"Then the nodes, in waves: one first, then twice as many each time a wave\n" +
|
||||
"comes back whole, up to --parallel at once. Each agent fetches the release\n" +
|
||||
"for its own platform, checks it against the release's checksums, swaps it\n" +
|
||||
"in and restarts; engines keep serving through all of it and are adopted\n" +
|
||||
"again, and no ssh into the nodes is needed.\n\n" +
|
||||
"A node counts as done when it is back on the new version, ready, with as\n" +
|
||||
"many ready replicas as before. One that is not within --timeout stops the\n" +
|
||||
"many ready replicas as before. A wave not back within --timeout stops the\n" +
|
||||
"upgrade there, so a bad release reaches one node rather than all of them.\n" +
|
||||
"Nodes already on the version are skipped, so running it again resumes.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
@@ -61,11 +68,28 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
|
||||
}
|
||||
}
|
||||
if !skipCP {
|
||||
if err := upgradeControlPlane(ctx, out, client, v, releases, serverSvc); err != nil {
|
||||
return err
|
||||
// The leader last, wherever this runs: every other control
|
||||
// plane is a standby, upgraded with nothing moving, and the
|
||||
// leader's restart hands over once, to a machine already on
|
||||
// the release.
|
||||
others := func() error {
|
||||
if !othersCP {
|
||||
return nil
|
||||
}
|
||||
return upgradeOtherControlPlanes(ctx, out, client, v, releases, sshUser)
|
||||
}
|
||||
self := func() error { return upgradeControlPlane(ctx, out, client, v, releases, serverSvc) }
|
||||
order := []func() error{others, self}
|
||||
if !thisLeads(ctx, client) {
|
||||
order = []func() error{self, others}
|
||||
}
|
||||
for _, step := range order {
|
||||
if err := step(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return upgradeNodes(ctx, out, client, v, nodes, perNode)
|
||||
return upgradeNodes(ctx, out, client, v, nodes, perNode, parallel)
|
||||
},
|
||||
}
|
||||
c.Flags().StringSliceVar(&nodes, "nodes", nil, "only these nodes (default: all)")
|
||||
@@ -74,11 +98,74 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
|
||||
"(default: as server.yaml says, the mirror's copy on an offline cluster, else the project's)")
|
||||
c.Flags().DurationVar(&perNode, "timeout", 3*time.Minute, "how long a node may take to come back")
|
||||
c.Flags().StringVar(&serverSvc, "service", "llmgw-server", "this machine's systemd unit")
|
||||
c.Flags().IntVar(¶llel, "parallel", 20, "the most nodes upgraded at once, once the first waves are back")
|
||||
c.Flags().StringVar(&sshUser, "ssh-user", os.Getenv("SUDO_USER"), "who to log in to the other control planes as")
|
||||
c.Flags().BoolVar(&othersCP, "other-control-planes", true, "upgrade the cluster's other control planes too, over ssh")
|
||||
return c
|
||||
}
|
||||
|
||||
// thisLeads says whether this machine's control plane leads; one that does
|
||||
// not say, alone or older, is taken to.
|
||||
func thisLeads(ctx context.Context, client func() *ctlClient) bool {
|
||||
var sum adminapi.Summary
|
||||
if err := client().get(ctx, "/api/v1/summary", &sum); err != nil {
|
||||
return true
|
||||
}
|
||||
for _, cp := range sum.ControlPlanes.Members {
|
||||
if cp.This {
|
||||
return cp.State == "leading"
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// upgradeOtherControlPlanes moves every other control plane of the cluster to
|
||||
// v, over ssh with `llmgw upgrade`. Nothing is changed here if one cannot be
|
||||
// reached.
|
||||
func upgradeOtherControlPlanes(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, user string) error {
|
||||
var sum adminapi.Summary
|
||||
if err := client().get(ctx, "/api/v1/summary", &sum); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, cp := range sum.ControlPlanes.Members {
|
||||
if cp.This {
|
||||
continue
|
||||
}
|
||||
r := &onboard.SSH{Addr: cp.Address, User: user}
|
||||
have, err := r.Run(ctx, "llmgw --version", nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s (%s) cannot be reached over ssh as %q: %v\n"+
|
||||
"run `sudo llmgw upgrade %s --service llmgw-server` there, then this again with "+
|
||||
"--other-control-planes=false", cp.Name, cp.Address, user, err, v)
|
||||
}
|
||||
if strings.Contains(string(have), " "+v+"\n") || strings.HasSuffix(strings.TrimSpace(string(have)), " "+v) {
|
||||
fmt.Fprintf(out, "== %s already runs %s\n", cp.Name, v)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(out, "== %s (%s): %s -> %s\n", cp.Name, cp.State, strings.TrimSpace(string(have)), v)
|
||||
res, err := r.Run(ctx, "sudo llmgw upgrade "+shellQuote(v)+" --service llmgw-server --releases "+shellQuote(releases), nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("upgrading %s: %v: %s", cp.Name, err, strings.TrimSpace(string(res)))
|
||||
}
|
||||
fmt.Fprintf(out, " upgraded and restarted\n")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// controlPlaneStep is what this machine needs to run v: the binary installed,
|
||||
// the service restarted, both, or neither. The binary is this command's own;
|
||||
// running is what the server says it runs, "" from one too old to say. A
|
||||
// binary put in place by hand is not a server running it.
|
||||
func controlPlaneStep(binary, running, v string) (install, restart bool) {
|
||||
install = binary != v
|
||||
return install, install || running != v
|
||||
}
|
||||
|
||||
func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, svc string) error {
|
||||
if version.Version == v {
|
||||
var sum adminapi.Summary
|
||||
_ = client().get(ctx, "/api/v1/summary", &sum)
|
||||
install, restart := controlPlaneStep(version.Version, sum.Version, v)
|
||||
if !restart {
|
||||
fmt.Fprintf(out, "this machine already runs %s\n", v)
|
||||
return nil
|
||||
}
|
||||
@@ -90,12 +177,20 @@ func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlC
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
|
||||
backup, err := release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
|
||||
if err != nil {
|
||||
return err
|
||||
backup := "the binary installed by hand before it"
|
||||
if install {
|
||||
fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
|
||||
if backup, err = release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, " installed, checksum verified; the old binary is %s\n", backup)
|
||||
} else {
|
||||
running := sum.Version
|
||||
if running == "" {
|
||||
running = "an older release"
|
||||
}
|
||||
fmt.Fprintf(out, "== this machine: %s is installed, and %s still runs it; restarting %s\n", v, running, svc)
|
||||
}
|
||||
fmt.Fprintf(out, " installed, checksum verified; the old binary is %s\n", backup)
|
||||
var before []adminapi.NodeView
|
||||
_ = client().get(ctx, "/api/v1/nodes", &before)
|
||||
if b, err := exec.CommandContext(ctx, "systemctl", "restart", svc).CombinedOutput(); err != nil { //nolint:gosec // fixed command
|
||||
@@ -133,13 +228,23 @@ func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlC
|
||||
}
|
||||
}
|
||||
|
||||
func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient, v string, only []string, perNode time.Duration) error {
|
||||
// upgradeNodes moves nodes to v in waves: one node first, then twice as
|
||||
// many each time one comes back whole, up to parallel at once.
|
||||
//
|
||||
// Waves rather than one at a time, because one at a time is hours on a
|
||||
// fleet of hundreds, and an agent's restart costs nothing served — the
|
||||
// engines keep running and are adopted again. One first, because a release
|
||||
// that does not come back should reach one node rather than a wave of them.
|
||||
func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient, v string, only []string,
|
||||
perNode time.Duration, parallel int,
|
||||
) error {
|
||||
var nodes []adminapi.NodeView
|
||||
if err := client().get(ctx, "/api/v1/nodes", &nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
|
||||
done, skipped := 0, 0
|
||||
var todo []adminapi.NodeView
|
||||
skipped := 0
|
||||
for _, n := range nodes {
|
||||
if len(only) > 0 && !slices.Contains(only, n.ID) {
|
||||
continue
|
||||
@@ -148,41 +253,21 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
start := time.Now()
|
||||
before := readyReplicasOn(ctx, client, n.ID)
|
||||
fmt.Fprintf(out, "== %s: %s -> %s\n", n.ID, n.AgentVersion, v)
|
||||
body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
|
||||
if err := client().post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
|
||||
return fmt.Errorf("%s: %w\nstopped here; `llmgw ctl logs %s` says more", n.ID, err, n.ID)
|
||||
}
|
||||
fmt.Fprintf(out, " new binary in place; waiting for it to come back with its %d replica(s)\n", before)
|
||||
deadline := time.Now().Add(perNode)
|
||||
for {
|
||||
var now []adminapi.NodeView
|
||||
_ = client().get(ctx, "/api/v1/nodes", &now)
|
||||
var cur adminapi.NodeView
|
||||
for _, x := range now {
|
||||
if x.ID == n.ID {
|
||||
cur = x
|
||||
}
|
||||
}
|
||||
after := readyReplicasOn(ctx, client, n.ID)
|
||||
if cur.AgentVersion == v && cur.State == "ready" && after >= before {
|
||||
fmt.Fprintf(out, " done in %s: ready, replicas %d/%d\n",
|
||||
time.Since(start).Round(time.Second), after, before)
|
||||
done++
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("%s did not come back within %s (version %q, state %q, replicas %d/%d)\n"+
|
||||
"stopped here, before the next node; `llmgw ctl logs %s` says why, and the node "+
|
||||
"keeps its previous binary beside the new one", n.ID, perNode, cur.AgentVersion,
|
||||
cur.State, after, before, n.ID)
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
todo = append(todo, n)
|
||||
}
|
||||
fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
|
||||
parallel = max(parallel, 1)
|
||||
start, size, done := time.Now(), 1, 0
|
||||
for len(todo) > 0 {
|
||||
wave := todo[:min(size, len(todo))]
|
||||
todo = todo[len(wave):]
|
||||
if err := upgradeWave(ctx, out, client, v, wave, perNode); err != nil {
|
||||
return fmt.Errorf("%w\nstopped after this wave: %d upgraded, %d not tried", err, done, len(todo))
|
||||
}
|
||||
done += len(wave)
|
||||
size = min(size*2, parallel)
|
||||
}
|
||||
fmt.Fprintf(out, "%d node(s) upgraded to %s in %s, %d already on it\n",
|
||||
done, v, time.Since(start).Round(time.Second), skipped)
|
||||
// The one machine this does not reach. Said every time, because it is
|
||||
// exactly the kind of thing that stays three versions behind unnoticed.
|
||||
fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
|
||||
@@ -190,23 +275,90 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
||||
return nil
|
||||
}
|
||||
|
||||
// readyReplicasOn counts the ready replicas with a shard on a node.
|
||||
func readyReplicasOn(ctx context.Context, client func() *ctlClient, node string) int {
|
||||
var reps []adminapi.ReplicaView
|
||||
if err := client().get(ctx, "/api/v1/replicas", &reps); err != nil {
|
||||
return 0
|
||||
// upgradeWave tells every node in the wave at once, and waits until each is
|
||||
// back on v, ready, with as many ready replicas as it had.
|
||||
func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v string,
|
||||
wave []adminapi.NodeView, perNode time.Duration,
|
||||
) error {
|
||||
start := time.Now()
|
||||
before := readyReplicasByNode(ctx, client)
|
||||
ids := make([]string, len(wave))
|
||||
for i, n := range wave {
|
||||
ids[i] = n.ID
|
||||
}
|
||||
fmt.Fprintf(out, "== %d node(s): %s\n", len(wave), abbreviate(ids, 8))
|
||||
body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
|
||||
var failed []string
|
||||
for _, n := range wave {
|
||||
// The control plane waits for the agent to fetch and check the
|
||||
// release, up to five minutes; fifteen seconds gave up on a node
|
||||
// that was upgrading.
|
||||
if err := client().patient(6*time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
|
||||
failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
|
||||
}
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("not told to upgrade:\n %s\n`llmgw ctl logs NODE` says more", strings.Join(failed, "\n "))
|
||||
}
|
||||
waiting := slices.Clone(ids)
|
||||
deadline := time.Now().Add(perNode)
|
||||
for {
|
||||
var now []adminapi.NodeView
|
||||
_ = client().get(ctx, "/api/v1/nodes", &now)
|
||||
after := readyReplicasByNode(ctx, client)
|
||||
state := map[string]adminapi.NodeView{}
|
||||
for _, x := range now {
|
||||
state[x.ID] = x
|
||||
}
|
||||
waiting = slices.DeleteFunc(waiting, func(id string) bool {
|
||||
cur := state[id]
|
||||
return cur.AgentVersion == v && cur.State == "ready" && after[id] >= before[id]
|
||||
})
|
||||
if len(waiting) == 0 {
|
||||
fmt.Fprintf(out, " done in %s\n", time.Since(start).Round(time.Second))
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
var why []string
|
||||
for _, id := range waiting {
|
||||
cur := state[id]
|
||||
why = append(why, fmt.Sprintf("%s (version %q, state %q, replicas %d/%d)",
|
||||
id, cur.AgentVersion, cur.State, after[id], before[id]))
|
||||
}
|
||||
return fmt.Errorf("not back within %s:\n %s\n`llmgw ctl logs NODE` says why, and each keeps its "+
|
||||
"previous binary beside the new one", perNode, strings.Join(why, "\n "))
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
// abbreviate lists up to n names, and says how many more.
|
||||
func abbreviate(names []string, n int) string {
|
||||
if len(names) <= n {
|
||||
return strings.Join(names, ", ")
|
||||
}
|
||||
return strings.Join(names[:n], ", ") + fmt.Sprintf(" and %d more", len(names)-n)
|
||||
}
|
||||
|
||||
// readyReplicasByNode counts, for each node, the ready replicas with a
|
||||
// shard on it.
|
||||
func readyReplicasByNode(ctx context.Context, client func() *ctlClient) map[string]int {
|
||||
var reps []adminapi.ReplicaView
|
||||
out := map[string]int{}
|
||||
if err := client().get(ctx, "/api/v1/replicas", &reps); err != nil {
|
||||
return out
|
||||
}
|
||||
n := 0
|
||||
for _, r := range reps {
|
||||
if r.State != "ready" {
|
||||
continue
|
||||
}
|
||||
for _, s := range r.Shards {
|
||||
if s.Node == node {
|
||||
n++
|
||||
break
|
||||
seen := map[string]bool{}
|
||||
for _, sh := range r.Shards {
|
||||
if !seen[sh.Node] {
|
||||
seen[sh.Node] = true
|
||||
out[sh.Node]++
|
||||
}
|
||||
}
|
||||
}
|
||||
return n
|
||||
return out
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
@@ -57,32 +58,95 @@ func (f *fakeFleet) server(t *testing.T) *httptest.Server {
|
||||
return srv
|
||||
}
|
||||
|
||||
func TestAnUpgradeGoesNodeByNodeAndSkipsTheDone(t *testing.T) {
|
||||
func TestAnUpgradeSkipsTheNodesAlreadyDone(t *testing.T) {
|
||||
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.1", "c": "v1.0.0"}}
|
||||
srv := f.server(t)
|
||||
var out bytes.Buffer
|
||||
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second)
|
||||
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Join(f.told, ",") != "a,c" {
|
||||
t.Errorf("told %v; b was already on the version", f.told)
|
||||
}
|
||||
if !strings.Contains(out.String(), "2 node(s) upgraded to v1.0.1, 1 already on it") {
|
||||
if !strings.Contains(out.String(), "2 node(s) upgraded to v1.0.1 in") || !strings.Contains(out.String(), "1 already on it") {
|
||||
t.Errorf("output:\n%s", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUpgradeStopsAtTheFirstNodeThatDoesNotComeBack(t *testing.T) {
|
||||
func TestAnUpgradeStopsAtTheFirstWaveThatDoesNotComeBack(t *testing.T) {
|
||||
// A bad release reaches one node, not all of them.
|
||||
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0"}, stuck: map[string]bool{"a": true}}
|
||||
srv := f.server(t)
|
||||
var out bytes.Buffer
|
||||
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 3*time.Second)
|
||||
if err == nil || !strings.Contains(err.Error(), "a did not come back") {
|
||||
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 3*time.Second, 1)
|
||||
if err == nil || !strings.Contains(err.Error(), "not back within") || !strings.Contains(err.Error(), "a (version") {
|
||||
t.Fatalf("err %v", err)
|
||||
}
|
||||
if len(f.told) != 1 {
|
||||
t.Errorf("went on to %v after a node failed to come back", f.told)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUpgradeGoesInWavesThatGrowUpToTheLimit(t *testing.T) {
|
||||
// One first, then twice as many each time, up to --parallel: hundreds of
|
||||
// nodes in minutes rather than hours, and a bad release still on one.
|
||||
versions := map[string]string{}
|
||||
for i := range 12 {
|
||||
versions[fmt.Sprintf("n%02d", i)] = "v1.0.0"
|
||||
}
|
||||
f := &fakeFleet{versions: versions}
|
||||
srv := f.server(t)
|
||||
var out bytes.Buffer
|
||||
if err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second, 4); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var sizes []string
|
||||
for _, line := range strings.Split(out.String(), "\n") {
|
||||
if n, ok := strings.CutPrefix(line, "== "); ok {
|
||||
sizes = append(sizes, strings.Fields(n)[0])
|
||||
}
|
||||
}
|
||||
if got := strings.Join(sizes, ","); got != "1,2,4,4,1" {
|
||||
t.Errorf("waves %s:\n%s", got, out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUpgradeKnowsWhetherItRunsOnTheLeader(t *testing.T) {
|
||||
sum := adminapi.Summary{}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
_ = json.NewEncoder(w).Encode(sum)
|
||||
}))
|
||||
defer srv.Close()
|
||||
client := func() *ctlClient { return newCtlClient(srv.URL) }
|
||||
if !thisLeads(t.Context(), client) {
|
||||
t.Error("a control plane with no rival does not lead")
|
||||
}
|
||||
sum.ControlPlanes.Members = []adminapi.ControlPlane{{Name: "a", State: "leading"}, {Name: "b", State: "standing by", This: true}}
|
||||
if thisLeads(t.Context(), client) {
|
||||
t.Error("a standby took itself for the leader: it would upgrade the leader before itself")
|
||||
}
|
||||
sum.ControlPlanes.Members[0].This, sum.ControlPlanes.Members[1].This = true, false
|
||||
if !thisLeads(t.Context(), client) {
|
||||
t.Error("the leader took itself for a standby")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABinaryInstalledByHandIsNotAServerRunningIt(t *testing.T) {
|
||||
// The binary put in place with install, llmgw-server not restarted: the
|
||||
// upgrade said "already runs" and left the old server running.
|
||||
for _, c := range []struct {
|
||||
binary, running string
|
||||
install, restart bool
|
||||
}{
|
||||
{"v1.0.0", "v1.0.0", true, true}, // the usual: fetch it, restart
|
||||
{"v1.0.1", "v1.0.0", false, true}, // installed by hand: restart only
|
||||
{"v1.0.1", "", false, true}, // a server too old to say
|
||||
{"v1.0.1", "v1.0.1", false, false},
|
||||
} {
|
||||
install, restart := controlPlaneStep(c.binary, c.running, "v1.0.1")
|
||||
if install != c.install || restart != c.restart {
|
||||
t.Errorf("binary %s, running %q: install %v restart %v", c.binary, c.running, install, restart)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
)
|
||||
|
||||
// firewallStateFile is where a control plane writes what the cluster's
|
||||
// settings say of the firewall, for llmgw-firewall-sync to apply as root:
|
||||
// the server itself runs as llmgw and cannot change the ruleset.
|
||||
const firewallStateFile = "/var/lib/llmgw/firewall.json"
|
||||
|
||||
// firewallState is what is written there.
|
||||
type firewallState struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Networks []string `json:"networks,omitempty"`
|
||||
}
|
||||
|
||||
// writeFirewallState records the settings' firewall for this machine's sync
|
||||
// unit. Without the state directory — a server not run by systemd — there
|
||||
// is no unit to read it, and nothing is written.
|
||||
func writeFirewallState(cfg config.Server) error {
|
||||
if _, err := os.Stat("/var/lib/llmgw"); err != nil {
|
||||
return nil //nolint:nilerr // not a control plane set up by init
|
||||
}
|
||||
b, _ := json.Marshal(firewallState{Enabled: cfg.Firewall.Enabled, Networks: cfg.FirewallNetworks})
|
||||
if old, err := os.ReadFile(firewallStateFile); err == nil && string(old) == string(b) {
|
||||
return nil
|
||||
}
|
||||
tmp := firewallStateFile + ".new"
|
||||
if err := os.WriteFile(tmp, b, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, firewallStateFile)
|
||||
}
|
||||
|
||||
// The units that apply it: a path unit watching the file, and the oneshot it
|
||||
// starts.
|
||||
const (
|
||||
firewallSyncPathUnit = "llmgw-firewall-sync.path"
|
||||
firewallSyncUnitFile = `[Unit]
|
||||
Description=llmgw firewall: apply the cluster's settings on this control plane
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/bin/llmgw firewall sync
|
||||
`
|
||||
firewallSyncPathFile = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this control plane
|
||||
|
||||
[Path]
|
||||
PathChanged=` + firewallStateFile + `
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`
|
||||
)
|
||||
|
||||
func newFirewallCmd() *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "firewall",
|
||||
Short: "llmgw's firewall on this machine",
|
||||
Long: "Whether llmgw keeps a firewall is the cluster's setting firewall.enabled\n" +
|
||||
"(llmgw ctl config set firewall.enabled false). Nodes are told by the\n" +
|
||||
"control plane; a control plane writes it to " + firewallStateFile + ",\n" +
|
||||
"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
|
||||
}
|
||||
var install bool
|
||||
sync := &cobra.Command{
|
||||
Use: "sync",
|
||||
Short: "Apply what the cluster's settings say of the firewall, here (root)",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if os.Geteuid() != 0 {
|
||||
return errors.New("the firewall is changed as root: run it with sudo")
|
||||
}
|
||||
if install {
|
||||
if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
|
||||
if _, err := os.Stat(firewallStateFile); err != nil {
|
||||
return nil // the server writes it when it starts
|
||||
}
|
||||
}
|
||||
b, err := os.ReadFile(firewallStateFile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var st firewallState
|
||||
if err := json.Unmarshal(b, &st); err != nil {
|
||||
return fmt.Errorf("%s: %w", firewallStateFile, err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
|
||||
defer cancel()
|
||||
msg, err := syncFirewall(ctx, firewall.Local{}, st)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), msg)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
|
||||
c.AddCommand(sync)
|
||||
return c
|
||||
}
|
||||
|
||||
// installFirewallSync writes the sync units on a machine and has the path
|
||||
// unit follow the state file.
|
||||
func installFirewallSync(ctx context.Context, r firewall.Runner) error {
|
||||
body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
|
||||
out, err := r.Run(ctx, `set -e
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
cat > "$t/all"
|
||||
python3 - "$t" <<'PY'
|
||||
import sys
|
||||
d = sys.argv[1]
|
||||
a, b = open(d + "/all", "rb").read().split(b"\x00")
|
||||
open(d + "/service", "wb").write(a)
|
||||
open(d + "/path", "wb").write(b)
|
||||
PY
|
||||
sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-sync.service
|
||||
sudo install -m 644 "$t/path" /etc/systemd/system/`+firewallSyncPathUnit+`
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now `+firewallSyncPathUnit+` >/dev/null 2>&1`, []byte(body))
|
||||
if err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", firewallSyncPathUnit, err, string(out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// syncFirewall puts a firewall state in force on this machine.
|
||||
func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (string, error) {
|
||||
if !st.Enabled {
|
||||
had, err := firewall.Disable(ctx, r)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if had {
|
||||
return "llmgw's firewall is off here", nil
|
||||
}
|
||||
return "llmgw's firewall was already off here", nil
|
||||
}
|
||||
loaded, err := firewall.Enable(ctx, r)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !loaded {
|
||||
return "no llmgw rules on this machine to put back; 'llmgw server init' writes them", nil
|
||||
}
|
||||
var extra []netip.Prefix
|
||||
for _, n := range st.Networks {
|
||||
if p, err := netip.ParsePrefix(n); err == nil {
|
||||
extra = append(extra, p)
|
||||
} else if a, err := netip.ParseAddr(n); err == nil {
|
||||
extra = append(extra, netip.PrefixFrom(a, a.BitLen()))
|
||||
}
|
||||
}
|
||||
if len(extra) > 0 {
|
||||
if _, err := firewall.Install(ctx, r, firewall.Spec{Allow: extra}); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
return "llmgw's firewall is on here", nil
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type scriptLog struct {
|
||||
scripts []string
|
||||
reply func(script string) string
|
||||
}
|
||||
|
||||
func (s *scriptLog) Run(_ context.Context, script string, _ []byte) ([]byte, error) {
|
||||
s.scripts = append(s.scripts, script)
|
||||
if s.reply != nil {
|
||||
return []byte(s.reply(script)), nil
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
|
||||
off := &scriptLog{}
|
||||
if _, err := syncFirewall(context.Background(), off, firewallState{Enabled: false}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(strings.Join(off.scripts, "\n"), "nft delete table inet llmgw") {
|
||||
t.Errorf("off did not take the table out: %q", off.scripts)
|
||||
}
|
||||
|
||||
none := &scriptLog{reply: func(s string) string {
|
||||
if strings.Contains(s, "echo loaded") {
|
||||
return "none\n"
|
||||
}
|
||||
return ""
|
||||
}}
|
||||
msg, err := syncFirewall(context.Background(), none, firewallState{Enabled: true})
|
||||
if err != nil || !strings.Contains(msg, "server init") {
|
||||
t.Errorf("on, with no rules here: %q %v", msg, err)
|
||||
}
|
||||
|
||||
on := &scriptLog{reply: func(s string) string {
|
||||
if strings.Contains(s, "echo loaded") {
|
||||
return "loaded\n"
|
||||
}
|
||||
return ""
|
||||
}}
|
||||
if msg, err := syncFirewall(context.Background(), on, firewallState{Enabled: true}); err != nil || msg != "llmgw's firewall is on here" {
|
||||
t.Errorf("on: %q %v", msg, err)
|
||||
}
|
||||
}
|
||||
@@ -14,7 +14,8 @@ func newGuardCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "guard",
|
||||
Short: "Guardrail policies: which apply where",
|
||||
Long: "Policies are defined in server.yaml under guard.policies — what requests and\n" +
|
||||
Long: "Policies are defined in the cluster's settings under guard.policies\n" +
|
||||
"(llmgw ctl config edit; in force everywhere within seconds) — what requests and\n" +
|
||||
"answers are checked for (personal data, secrets, prompt injection, patterns,\n" +
|
||||
"tools) and what is done: flag, redact or block. guard.default applies to\n" +
|
||||
"every request; attach others to an organization, team, project, user or\n" +
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"golang.org/x/net/http2"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/clientip"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
// forwardPrefix is where the leader answers a standby's admin API, on the
|
||||
// agent port: the one port control planes already reach each other on, with
|
||||
// the cluster's certificates both ways. The admin port itself listens on
|
||||
// loopback, and should — it serves pprof.
|
||||
const forwardPrefix = "/_admin"
|
||||
|
||||
// Headers a forwarded request and its answer carry: which standby asked, and
|
||||
// which control plane answered.
|
||||
const (
|
||||
hdrForwardedBy = "X-Llmgw-Forwarded-By"
|
||||
hdrAnsweredBy = "X-Llmgw-Answered-By"
|
||||
)
|
||||
|
||||
// leaderForward sends a standby's admin API to the leader.
|
||||
//
|
||||
// Everything the console and ctl show is the leader's: nodes, replicas and
|
||||
// jobs live in its memory, put there by the agent streams only it holds. A
|
||||
// standby answering from its own would show an empty cluster, and accept
|
||||
// changes it cannot carry out. So a standby answers nothing under /api/
|
||||
// itself; the leader does, and the console looks the same on either machine.
|
||||
type leaderForward struct {
|
||||
st *store.DB
|
||||
lead interface{ IsLeader() bool }
|
||||
self string
|
||||
tls tlsconf.Config
|
||||
log *slog.Logger
|
||||
}
|
||||
|
||||
// standby wraps the admin handler: /api/ goes to the leader while this
|
||||
// process is not it.
|
||||
func (f *leaderForward) standby(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if adminStaysHere(r.URL.Path) || f.lead.IsLeader() {
|
||||
next.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
target, name, err := f.leader(r.Context())
|
||||
if err != nil {
|
||||
writeForwardError(w, http.StatusServiceUnavailable,
|
||||
"this control plane is standing by, and cannot reach the leader: "+err.Error())
|
||||
return
|
||||
}
|
||||
tc, err := f.tls.ClientConfig(target.Hostname())
|
||||
if err != nil {
|
||||
writeForwardError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
p := &httputil.ReverseProxy{
|
||||
Rewrite: func(pr *httputil.ProxyRequest) {
|
||||
pr.Out.URL.Scheme, pr.Out.URL.Host = "https", target.Host
|
||||
pr.Out.URL.Path = forwardPrefix + pr.In.URL.Path
|
||||
pr.Out.URL.RawPath = ""
|
||||
pr.Out.Host = target.Host
|
||||
// The leader's own token is not this one's, and the
|
||||
// certificate is what it believes.
|
||||
pr.Out.Header.Del("Authorization")
|
||||
pr.Out.Header.Set(hdrForwardedBy, f.self)
|
||||
},
|
||||
Transport: &http2.Transport{TLSClientConfig: tc},
|
||||
// Streams, such as a job's progress, pass through as they come.
|
||||
FlushInterval: -1,
|
||||
ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) {
|
||||
writeForwardError(w, http.StatusBadGateway, fmt.Sprintf("the leader, %s, did not answer: %v", name, err))
|
||||
},
|
||||
}
|
||||
p.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// adminStaysHere are the admin paths about this process rather than the
|
||||
// cluster: its profile, its own metrics, its own health. Never forwarded, and
|
||||
// never answered for another — pprof dumps the heap, keys and all.
|
||||
func adminStaysHere(path string) bool {
|
||||
return strings.HasPrefix(path, "/debug/") || path == "/metrics" || path == "/health"
|
||||
}
|
||||
|
||||
// publicStaysHere are the client port's paths that must describe this
|
||||
// process: a load balancer and keepalived ask them which machine to send
|
||||
// traffic to, and a standby saying the leader is ready would draw it.
|
||||
func publicStaysHere(path string) bool {
|
||||
return path == "/health" || path == "/ready" || path == "/leader" || path == "/metrics"
|
||||
}
|
||||
|
||||
// public wraps the client-facing handler: on a standby, everything but its
|
||||
// own health goes to the leader's client port — the portal, its admin pages
|
||||
// and the API, which on a standby would show an empty cluster and serve
|
||||
// nothing. Plain HTTP, as the port is; the leader trusts a control plane's
|
||||
// X-Forwarded-For, so a client is still known by its own address.
|
||||
func (f *leaderForward) public(next http.Handler, port string) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
// Once only: a request another control plane forwarded is answered
|
||||
// here, leading or not, rather than sent on while the lease moves.
|
||||
if publicStaysHere(r.URL.Path) || f.lead.IsLeader() || r.Header.Get(hdrForwardedBy) != "" {
|
||||
next.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
target, _, err := f.leader(r.Context())
|
||||
if err != nil || target.Hostname() == "" {
|
||||
next.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
host := net.JoinHostPort(target.Hostname(), port)
|
||||
p := &httputil.ReverseProxy{
|
||||
Rewrite: func(pr *httputil.ProxyRequest) {
|
||||
pr.Out.URL.Scheme, pr.Out.URL.Host = "http", host
|
||||
// The name the client used, for the portal's links and cookies.
|
||||
pr.Out.Host = pr.In.Host
|
||||
peer, _, _ := net.SplitHostPort(pr.In.RemoteAddr)
|
||||
xff := pr.In.Header.Get("X-Forwarded-For")
|
||||
if xff != "" {
|
||||
xff += ", "
|
||||
}
|
||||
pr.Out.Header.Set("X-Forwarded-For", xff+peer)
|
||||
for _, h := range []string{"X-Forwarded-Proto", "X-Forwarded-Host"} {
|
||||
if v := pr.In.Header.Get(h); v != "" {
|
||||
pr.Out.Header.Set(h, v)
|
||||
}
|
||||
}
|
||||
pr.Out.Header.Set(hdrForwardedBy, f.self)
|
||||
},
|
||||
FlushInterval: -1,
|
||||
ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) {
|
||||
writeForwardError(w, http.StatusBadGateway, "this control plane is standing by, and the leader did not answer: "+err.Error())
|
||||
},
|
||||
}
|
||||
p.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// trustControlPlanes keeps the client port trusting the other control
|
||||
// planes' X-Forwarded-For, besides router.trusted_proxies: a standby passes
|
||||
// requests on, and the client is the address it was given, not the standby.
|
||||
// Read again every half minute, for one added since.
|
||||
func (f *leaderForward) trustControlPlanes(ctx context.Context, configured []string) {
|
||||
for {
|
||||
list := slices.Clone(configured)
|
||||
if cps, err := f.st.ControlPlanes(ctx); err == nil {
|
||||
for _, cp := range cps {
|
||||
if u, err := url.Parse(cp.URL); err == nil && u.Hostname() != "" {
|
||||
list = append(list, u.Hostname())
|
||||
}
|
||||
}
|
||||
}
|
||||
clientip.SetTrusted(list)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(30 * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// leader is where the lease's holder is reached, and its name.
|
||||
func (f *leaderForward) leader(ctx context.Context) (*url.URL, string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
|
||||
defer cancel()
|
||||
lease, err := f.st.Leadership(ctx, control.ControlRole)
|
||||
if err != nil || lease.Owner == "" || time.Now().After(lease.ExpiresAt) {
|
||||
return nil, "", errors.New("no control plane holds the lease right now")
|
||||
}
|
||||
name := control.HolderName(lease.Owner)
|
||||
if name == f.self {
|
||||
// This machine's previous process, whose lease has not run out.
|
||||
return nil, name, errors.New("the lease is still held by this machine's previous run; it runs out within seconds")
|
||||
}
|
||||
cps, err := f.st.ControlPlanes(ctx)
|
||||
if err != nil {
|
||||
return nil, name, err
|
||||
}
|
||||
for _, cp := range cps {
|
||||
if cp.Name == name {
|
||||
u, err := url.Parse(cp.URL)
|
||||
return u, name, err
|
||||
}
|
||||
}
|
||||
return nil, name, fmt.Errorf("%s leads, and is not among the control planes the store lists", name)
|
||||
}
|
||||
|
||||
// leaderSide answers forwarded requests, on the agent port: only from another
|
||||
// control plane of this cluster, only while this one leads, and only the API.
|
||||
func (f *leaderForward) leaderSide(admin http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
path := strings.TrimPrefix(r.URL.Path, forwardPrefix)
|
||||
if adminStaysHere(path) {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
if err := f.fromControlPlane(r); err != nil {
|
||||
writeForwardError(w, http.StatusForbidden, err.Error())
|
||||
return
|
||||
}
|
||||
if !f.lead.IsLeader() {
|
||||
// Not passed on again: two standbys must not bounce a request
|
||||
// between them while the lease moves.
|
||||
writeForwardError(w, http.StatusServiceUnavailable, "this control plane no longer leads; ask again")
|
||||
return
|
||||
}
|
||||
r2 := r.Clone(r.Context())
|
||||
r2.URL.Path, r2.URL.RawPath = path, ""
|
||||
w.Header().Set(hdrAnsweredBy, f.self)
|
||||
admin.ServeHTTP(w, r2)
|
||||
})
|
||||
}
|
||||
|
||||
// fromControlPlane checks the client certificate names a control plane the
|
||||
// store lists. A node's certificate is from the same authority, and a node
|
||||
// must not reach the admin API by presenting it.
|
||||
func (f *leaderForward) fromControlPlane(r *http.Request) error {
|
||||
if r.TLS == nil || len(r.TLS.VerifiedChains) == 0 {
|
||||
return errors.New("a control plane's certificate is required")
|
||||
}
|
||||
cert := r.TLS.VerifiedChains[0][0]
|
||||
if !slices.Contains(cert.ExtKeyUsage, x509.ExtKeyUsageServerAuth) {
|
||||
return errors.New("not a control plane's certificate")
|
||||
}
|
||||
cps, err := f.st.ControlPlanes(r.Context())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, cp := range cps {
|
||||
if cp.Name == cert.Subject.CommonName {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("%s is not a control plane of this cluster", cert.Subject.CommonName)
|
||||
}
|
||||
|
||||
func writeForwardError(w http.ResponseWriter, code int, msg string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(code)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": msg})
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
type leading struct{ v atomic.Bool }
|
||||
|
||||
func (l *leading) IsLeader() bool { return l.v.Load() }
|
||||
|
||||
// forwardCluster is two control planes of one cluster on one store: a
|
||||
// leader whose agent port answers forwarded admin requests, and a standby.
|
||||
type forwardCluster struct {
|
||||
st *store.DB
|
||||
dir string
|
||||
leaderLeads *leading
|
||||
standby, leader *leaderForward
|
||||
leaderURL string
|
||||
}
|
||||
|
||||
func newForwardCluster(t *testing.T) *forwardCluster {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
ca, err := tlsconf.NewCA("test CA")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := ca.Write(dir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, name := range []string{"ctl-a", "ctl-b"} {
|
||||
leaf, err := ca.IssuePeer(name, []string{"127.0.0.1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := leaf.Write(dir, name); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
node, err := ca.IssueClient("node1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := node.Write(dir, "node1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfgFor := func(name string) tlsconf.Config {
|
||||
return tlsconf.Config{CA: filepath.Join(dir, "ca.pem"), Cert: filepath.Join(dir, name+".pem"), Key: filepath.Join(dir, name+"-key.pem")}
|
||||
}
|
||||
|
||||
st, err := store.Open(t.Context(), filepath.Join(dir, "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
|
||||
c := &forwardCluster{st: st, dir: dir, leaderLeads: &leading{}}
|
||||
c.leaderLeads.v.Store(true)
|
||||
log := slog.New(slog.DiscardHandler)
|
||||
c.leader = &leaderForward{st: st, lead: c.leaderLeads, self: "ctl-a", tls: cfgFor("ctl-a"), log: log}
|
||||
standbyLeads := &leading{}
|
||||
c.standby = &leaderForward{st: st, lead: standbyLeads, self: "ctl-b", tls: cfgFor("ctl-b"), log: log}
|
||||
|
||||
// The leader's admin API: says whose it is, and who it was asked by.
|
||||
admin := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"path": r.URL.Path, "by": r.Header.Get(hdrForwardedBy),
|
||||
"auth": r.Header.Get("Authorization"), "actor": r.Header.Get("X-Llmgw-Actor")})
|
||||
})
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle(forwardPrefix+"/", c.leader.leaderSide(admin))
|
||||
srv := httptest.NewUnstartedServer(mux)
|
||||
tc, err := cfgFor("ctl-a").ServerConfig()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.TLS = tc
|
||||
srv.EnableHTTP2 = true
|
||||
srv.StartTLS()
|
||||
t.Cleanup(srv.Close)
|
||||
c.leaderURL = srv.URL
|
||||
|
||||
for _, cp := range []store.ControlPlane{{URL: srv.URL, Name: "ctl-a"}, {URL: "https://127.0.0.1:1", Name: "ctl-b"}} {
|
||||
if err := st.PutControlPlane(t.Context(), cp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, err := st.AcquireLeadership(t.Context(), control.ControlRole, "ctl-a-0123456789abcdef", time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// ask sends one request through the standby's admin API.
|
||||
func (c *forwardCluster) ask(t *testing.T, path string) (*http.Response, map[string]string) {
|
||||
t.Helper()
|
||||
local := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"path": "local"})
|
||||
})
|
||||
srv := httptest.NewServer(c.standby.standby(local))
|
||||
defer srv.Close()
|
||||
req, _ := http.NewRequestWithContext(t.Context(), http.MethodGet, srv.URL+path, nil)
|
||||
req.Header.Set("Authorization", "Bearer the-standbys-own-token")
|
||||
req.Header.Set("X-Llmgw-Actor", "root@ctl-b")
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
var got map[string]string
|
||||
_ = json.Unmarshal(b, &got)
|
||||
return resp, got
|
||||
}
|
||||
|
||||
func TestAStandbysAdminAPIIsTheLeaders(t *testing.T) {
|
||||
c := newForwardCluster(t)
|
||||
resp, got := c.ask(t, "/api/v1/summary")
|
||||
if resp.StatusCode != http.StatusOK || got["path"] != "/api/v1/summary" {
|
||||
t.Fatalf("%d %v", resp.StatusCode, got)
|
||||
}
|
||||
if resp.Header.Get(hdrAnsweredBy) != "ctl-a" || got["by"] != "ctl-b" {
|
||||
t.Errorf("answered by %q, asked by %q", resp.Header.Get(hdrAnsweredBy), got["by"])
|
||||
}
|
||||
if got["auth"] != "" {
|
||||
t.Error("the standby's token was passed on to the leader")
|
||||
}
|
||||
if got["actor"] != "root@ctl-b" {
|
||||
t.Errorf("who asked was lost: %q", got["actor"])
|
||||
}
|
||||
// The dashboard's pages are the leader's too.
|
||||
if _, got := c.ask(t, "/nodes/cuda01"); got["path"] != "/nodes/cuda01" {
|
||||
t.Errorf("a dashboard page was answered here: %v", got)
|
||||
}
|
||||
// What is about this process stays on it: pprof, its metrics.
|
||||
for _, p := range []string{"/debug/pprof/", "/metrics"} {
|
||||
if _, got := c.ask(t, p); got["path"] != "local" {
|
||||
t.Errorf("%s was forwarded: %v", p, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheLeaderAnswersOnlyControlPlanesAndOnlyWhileItLeads(t *testing.T) {
|
||||
c := newForwardCluster(t)
|
||||
call := func(cert string) int {
|
||||
cfg := tlsconf.Config{CA: filepath.Join(c.dir, "ca.pem"), Cert: filepath.Join(c.dir, cert+".pem"), Key: filepath.Join(c.dir, cert+"-key.pem")}
|
||||
tc, err := cfg.ClientConfig("127.0.0.1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tc.NextProtos = nil
|
||||
client := &http.Client{Transport: &http.Transport{TLSClientConfig: tc}}
|
||||
resp, err := client.Get(c.leaderURL + forwardPrefix + "/api/v1/summary")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
return resp.StatusCode
|
||||
}
|
||||
if code := call("ctl-b"); code != http.StatusOK {
|
||||
t.Errorf("a control plane was refused: %d", code)
|
||||
}
|
||||
// A node's certificate is from the same authority, and is no way in.
|
||||
if code := call("node1"); code != http.StatusForbidden {
|
||||
t.Errorf("a node reached the admin API: %d", code)
|
||||
}
|
||||
c.leaderLeads.v.Store(false)
|
||||
if code := call("ctl-b"); code != http.StatusServiceUnavailable {
|
||||
t.Errorf("a control plane that no longer leads answered: %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStandbyWithNoLeaderSaysSo(t *testing.T) {
|
||||
c := newForwardCluster(t)
|
||||
if err := c.st.ReleaseLeadership(t.Context(), control.ControlRole, "ctl-a-0123456789abcdef"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp, got := c.ask(t, "/api/v1/summary")
|
||||
if resp.StatusCode != http.StatusServiceUnavailable || !strings.Contains(got["error"], "standing by") {
|
||||
t.Errorf("%d %v", resp.StatusCode, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStandbysPortalIsTheLeaders(t *testing.T) {
|
||||
c := newForwardCluster(t)
|
||||
// The leader's client port, on the address the store lists for it.
|
||||
var got http.Header
|
||||
var gotHost string
|
||||
leader := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got, gotHost = r.Header.Clone(), r.Host
|
||||
_, _ = w.Write([]byte("leader " + r.URL.Path))
|
||||
}))
|
||||
defer leader.Close()
|
||||
_, port, _ := net.SplitHostPort(strings.TrimPrefix(leader.URL, "http://"))
|
||||
|
||||
local := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("standby " + r.URL.Path)) })
|
||||
srv := httptest.NewServer(c.standby.public(local, port))
|
||||
defer srv.Close()
|
||||
get := func(path string, hdr map[string]string) string {
|
||||
req, _ := http.NewRequestWithContext(t.Context(), http.MethodGet, srv.URL+path, nil)
|
||||
req.Host = "llmgw.example"
|
||||
for k, v := range hdr {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
if body := get("/admin/replicas", map[string]string{"X-Forwarded-For": "203.0.113.7", "X-Forwarded-Proto": "https"}); body != "leader /admin/replicas" {
|
||||
t.Fatalf("the portal was answered by %q", body)
|
||||
}
|
||||
if gotHost != "llmgw.example" || got.Get("X-Forwarded-Proto") != "https" ||
|
||||
!strings.HasPrefix(got.Get("X-Forwarded-For"), "203.0.113.7, ") || got.Get(hdrForwardedBy) != "ctl-b" {
|
||||
t.Errorf("host %q, headers %v", gotHost, got)
|
||||
}
|
||||
// What a load balancer asks is about this machine.
|
||||
for _, p := range []string{"/ready", "/health", "/leader"} {
|
||||
if body := get(p, nil); body != "standby "+p {
|
||||
t.Errorf("%s answered by %q", p, body)
|
||||
}
|
||||
}
|
||||
// Forwarded once only.
|
||||
if body := get("/v1/models", map[string]string{hdrForwardedBy: "ctl-c"}); body != "standby /v1/models" {
|
||||
t.Errorf("a forwarded request was forwarded again: %q", body)
|
||||
}
|
||||
}
|
||||
@@ -89,10 +89,7 @@ func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPla
|
||||
if err := in.chown(dir, true); err != nil {
|
||||
return err
|
||||
}
|
||||
addr := ""
|
||||
if ips := localAddrs(); len(ips) > 0 {
|
||||
addr = ips[0].String()
|
||||
}
|
||||
addr := primaryAddr()
|
||||
if base == "" {
|
||||
if addr == "" {
|
||||
return errors.New("this machine has no address to give nodes; give one with --base")
|
||||
|
||||
@@ -73,6 +73,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
||||
f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
|
||||
f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
|
||||
f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
|
||||
f.BoolVar(&req.Tools, "tools", false,
|
||||
"say it makes tool calls (for agents and editors); check a running one with llmgw ctl tool-check")
|
||||
f.StringVar(&req.GuardFormat, "guard-format", "",
|
||||
"serve it as the guardrails' judge (role: guard), read as llamaguard3, shieldgemma or json; hidden from clients")
|
||||
f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
|
||||
|
||||
+55
-4
@@ -5,6 +5,8 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/netip"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -16,7 +18,9 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
@@ -164,23 +168,36 @@ func (na *nodeAdd) prepare() error {
|
||||
// are not known until the node says what it has, and are not applied
|
||||
// here; doctor's verdict on vLLM is a warning either way.
|
||||
if o.VLLMVersion == "" {
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil {
|
||||
if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil {
|
||||
o.VLLMVersion = cfg.WantedEngines()["vllm"]
|
||||
if o.VLLMVersion == "" {
|
||||
o.VLLMVersion = "none"
|
||||
}
|
||||
}
|
||||
}
|
||||
// The node's ports — its engines', its metrics — for the cluster only.
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil && !o.NoFirewall {
|
||||
// The node's ports — its engines', its metrics — for the cluster only,
|
||||
// unless the cluster's settings turned llmgw's firewall off.
|
||||
if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && !cfg.Firewall.Enabled {
|
||||
o.NoFirewall = true
|
||||
} else if err == nil && !o.NoFirewall {
|
||||
from, to := cfg.EnginePorts()
|
||||
o.FirewallPorts = []firewall.Port{firewall.Engines(from, to), firewall.NodeExporter}
|
||||
o.FirewallAllow, _ = cfg.FirewallAllow()
|
||||
}
|
||||
// Where the mirror serves model files, for the node to be asked to reach.
|
||||
if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && cfg.Mirror.API != "" {
|
||||
if mc, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 5*time.Second); err == nil {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
if list, err := mc.List(ctx); err == nil {
|
||||
o.MirrorBase = list.Base
|
||||
}
|
||||
cancel()
|
||||
}
|
||||
}
|
||||
// Where the control plane upgrades nodes from, so a node joins on the
|
||||
// same release from the same place: an offline cluster's mirror.
|
||||
if o.Releases == "" {
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil {
|
||||
if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil {
|
||||
o.Releases = cfg.ReleasesBase()
|
||||
}
|
||||
if o.Releases == "" {
|
||||
@@ -266,6 +283,9 @@ func (na *nodeAdd) run(ctx context.Context, addrs []string, out io.Writer) error
|
||||
if len(addrs) > 1 {
|
||||
title += fmt.Sprintf(" (%d of %d)", i+1, len(addrs))
|
||||
}
|
||||
if !o.NoFirewall {
|
||||
na.letIn(ctx, o.Addr, out)
|
||||
}
|
||||
var node string
|
||||
var err error
|
||||
if tui {
|
||||
@@ -297,6 +317,37 @@ func (na *nodeAdd) run(ctx context.Context, addrs []string, out io.Writer) error
|
||||
return nil
|
||||
}
|
||||
|
||||
// letIn opens this control plane's firewall to the node, when it has one
|
||||
// and the node is on a network it does not already let in: otherwise a node
|
||||
// on another subnet is set up and then never heard from. Said, not fatal.
|
||||
func (na *nodeAdd) letIn(ctx context.Context, host string, out io.Writer) {
|
||||
var addrs []string
|
||||
if a, err := netip.ParseAddr(host); err == nil {
|
||||
addrs = []string{a.String()}
|
||||
} else {
|
||||
ips, err := net.DefaultResolver.LookupNetIP(ctx, "ip4", host)
|
||||
if err != nil {
|
||||
fmt.Fprintf(out, " !! firewall: cannot resolve %s (%v); if it is on another network, "+
|
||||
"list it in firewall_networks and run 'llmgw server init'\n", host, err)
|
||||
return
|
||||
}
|
||||
for _, ip := range ips {
|
||||
addrs = append(addrs, ip.Unmap().String())
|
||||
}
|
||||
}
|
||||
added, err := firewall.Admit(ctx, pgha.Local{}, addrs)
|
||||
switch {
|
||||
case err != nil:
|
||||
fmt.Fprintf(out, " !! firewall: could not let %s in: %v\n", host, err)
|
||||
case len(added) > 0:
|
||||
shown := make([]string, len(added))
|
||||
for i, p := range added {
|
||||
shown[i] = p.String()
|
||||
}
|
||||
fmt.Fprintf(out, " ok firewall: this control plane now lets in %s\n", strings.Join(shown, ", "))
|
||||
}
|
||||
}
|
||||
|
||||
func (na *nodeAdd) runPlain(ctx context.Context, o onboard.Options, title string, out io.Writer) (string, error) {
|
||||
p := &plainProgress{out: out}
|
||||
fmt.Fprintf(out, "%s\n", title)
|
||||
|
||||
@@ -59,6 +59,8 @@ func Root() *cobra.Command {
|
||||
newSelfUpgradeCmd(),
|
||||
newReleaseCmd(),
|
||||
newJoinCmd(),
|
||||
newFirewallCmd(),
|
||||
newWipeCmd(),
|
||||
newDocsCmd(),
|
||||
)
|
||||
return root
|
||||
|
||||
+139
-6
@@ -16,6 +16,7 @@ import (
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -80,10 +81,13 @@ func newServerCmd() *cobra.Command {
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
log := logger()
|
||||
|
||||
cfg, err := config.Load(configPath)
|
||||
// server.yaml for this machine's own keys, the store's settings
|
||||
// for the cluster's: the same on every control plane.
|
||||
boot, err := loadClusterSettings(cmd.Context(), configPath, log)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg := boot.cfg
|
||||
if listen != "" {
|
||||
cfg.Listen = listen
|
||||
}
|
||||
@@ -166,7 +170,7 @@ func newServerCmd() *cobra.Command {
|
||||
}
|
||||
gate.SetJudge(rt.Judge, rt.GuardFormat)
|
||||
rt.SetGuard(gate)
|
||||
rt.SetGuardLibrary(cfg.Guard)
|
||||
rt.SetGuardLibrary(gate.Library)
|
||||
if err := rt.RegisterMetrics(gate); err != nil {
|
||||
return fmt.Errorf("guard metrics: %w", err)
|
||||
}
|
||||
@@ -175,6 +179,10 @@ func newServerCmd() *cobra.Command {
|
||||
|
||||
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
// Cancelled to start again on changed settings.
|
||||
ctx, cancelRun := context.WithCancel(ctx)
|
||||
defer cancelRun()
|
||||
var restartReason atomic.Pointer[string]
|
||||
|
||||
rt.Start(ctx)
|
||||
defer rt.Stop()
|
||||
@@ -215,6 +223,10 @@ func newServerCmd() *cobra.Command {
|
||||
// The control plane replaces the static endpoint list: agents
|
||||
// register, the reconciler places replicas, and Ready ones are
|
||||
// published to the router (plan §6.4).
|
||||
// The settings in force, for what reads them at each use.
|
||||
var current atomic.Pointer[config.Server]
|
||||
current.Store(&cfg)
|
||||
var agentSvc *agentsvc.Service
|
||||
var agentHandler http.Handler
|
||||
var agentPath string
|
||||
var admin *control.Controller
|
||||
@@ -239,6 +251,14 @@ func newServerCmd() *cobra.Command {
|
||||
return err
|
||||
}
|
||||
defer ctl.Close()
|
||||
agentSvc = svc
|
||||
// Whether nodes keep llmgw's firewall, as the settings say now.
|
||||
svc.SetFirewall(func() *llmgwv1.Firewall {
|
||||
c := current.Load()
|
||||
from, to := c.EnginePorts()
|
||||
return &llmgwv1.Firewall{Enabled: c.Firewall.Enabled, Networks: c.FirewallNetworks,
|
||||
EnginePortsFrom: int32(from), EnginePortsTo: int32(to)} //nolint:gosec // ports
|
||||
})
|
||||
ctl.Reconciler().SetNotifier(notifier)
|
||||
admin = ctl
|
||||
agentPath, agentHandler = svc.Handler()
|
||||
@@ -330,6 +350,9 @@ func newServerCmd() *cobra.Command {
|
||||
return out, err
|
||||
})
|
||||
}
|
||||
if st := rt.Store(); st != nil {
|
||||
api.SetControlPlanes(newControlPlaneProbe(st, cfg.Listen).Status)
|
||||
}
|
||||
adminAPI = api
|
||||
api.SetReleases(cfg.Releases)
|
||||
api.SetNotifier(notifier)
|
||||
@@ -338,12 +361,16 @@ func newServerCmd() *cobra.Command {
|
||||
evals = newEvalService(cfg, rt, admin, log)
|
||||
api.SetEval(evals)
|
||||
if st := rt.Store(); st != nil && gate != nil {
|
||||
api.SetGuard(&adminapi.GuardAdmin{Library: cfg.Guard, Store: st, Invalidate: gate.Invalidate})
|
||||
api.SetGuard(&adminapi.GuardAdmin{Library: gate.Library, Store: st, Invalidate: gate.Invalidate})
|
||||
}
|
||||
if st := rt.Store(); st != nil {
|
||||
// Removing a node revokes its certificate.
|
||||
api.SetRevoker(st.RevokeNode)
|
||||
api.SetAudit(st)
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
||||
Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
|
||||
}
|
||||
// Catalog changes need somewhere to persist, so this is wired
|
||||
// only when there is a store. Without one, the endpoints say so
|
||||
@@ -375,6 +402,17 @@ func newServerCmd() *cobra.Command {
|
||||
} else {
|
||||
admin.ReplaceCatalog(ctx, built)
|
||||
}
|
||||
// And again on becoming the leader: models added or
|
||||
// removed through the leader before are in the store.
|
||||
if l := admin.Leadership(); l != nil {
|
||||
l.OnLead(func() {
|
||||
if built, err := svc.Load(ctx); err != nil {
|
||||
log.Error("could not read the stored models on becoming the leader", "err", err)
|
||||
} else {
|
||||
admin.ReplaceCatalog(ctx, built)
|
||||
}
|
||||
})
|
||||
}
|
||||
api.SetModels(svc)
|
||||
catalogSvc = svc
|
||||
// And the mirror page, so a model that has just been
|
||||
@@ -499,8 +537,23 @@ func newServerCmd() *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
adminHandler := auditAPI(rt, adminMux, log)
|
||||
// With a rival, a standby's admin API is the leader's: what the
|
||||
// console and ctl show lives in the leader's memory.
|
||||
var forward *leaderForward
|
||||
if admin != nil && admin.Leadership() != nil && rt.Store() != nil && cfg.AgentTLS.Enabled() {
|
||||
self, _ := os.Hostname()
|
||||
forward = &leaderForward{st: rt.Store(), lead: admin.Leadership(), self: self, tls: cfg.AgentTLS, log: log}
|
||||
}
|
||||
served := adminHandler
|
||||
if forward != nil {
|
||||
served = forward.standby(adminHandler)
|
||||
_, port, _ := net.SplitHostPort(cfg.Listen)
|
||||
apiSrv.Handler = forward.public(apiSrv.Handler, port)
|
||||
go forward.trustControlPlanes(ctx, cfg.Router.TrustedProxies)
|
||||
}
|
||||
adminSrv := withLimits(&http.Server{
|
||||
Handler: router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, auditAPI(rt, adminMux, log))),
|
||||
Handler: router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, served)),
|
||||
ReadTimeout: time.Minute,
|
||||
})
|
||||
|
||||
@@ -566,6 +619,9 @@ func newServerCmd() *cobra.Command {
|
||||
if agentHandler != nil {
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle(agentPath, agentHandler)
|
||||
if forward != nil {
|
||||
mux.Handle(forwardPrefix+"/", router.AdminGuard(forward.leaderSide(adminHandler)))
|
||||
}
|
||||
// No read timeout: an agent's stream is one request that
|
||||
// lasts as long as the agent does.
|
||||
agentSrv = withLimits(&http.Server{Handler: mux})
|
||||
@@ -606,6 +662,59 @@ func newServerCmd() *cobra.Command {
|
||||
}
|
||||
}
|
||||
|
||||
// Every change to the settings in force: what can change live
|
||||
// does, the rest starts this process again, which systemd brings
|
||||
// back on them.
|
||||
// And this machine's own firewall, through its sync unit.
|
||||
if err := writeFirewallState(cfg); err != nil {
|
||||
log.Warn("could not record the firewall's settings for this machine", "err", err)
|
||||
}
|
||||
watch := &settingsWatch{path: configPath, st: rt.Store(), cur: boot, log: log,
|
||||
live: func(next config.Server, changed []string) bool {
|
||||
if !onlyLive(changed) {
|
||||
return false
|
||||
}
|
||||
current.Store(&next)
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "firewall") }) {
|
||||
if err := writeFirewallState(next); err != nil {
|
||||
log.Warn("could not record the firewall's settings for this machine", "err", err)
|
||||
}
|
||||
if agentSvc != nil {
|
||||
agentSvc.BroadcastFirewall()
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "guard") }) {
|
||||
if gate == nil {
|
||||
// Built at start, and only when there are policies.
|
||||
return len(next.Guard.Policies) == 0
|
||||
}
|
||||
if err := gate.Replace(next.Guard); err != nil {
|
||||
log.Error("the changed guard policies are not valid; the old ones stay in force", "err", err)
|
||||
}
|
||||
}
|
||||
return true
|
||||
},
|
||||
restart: func(reason string) {
|
||||
if os.Getenv("INVOCATION_ID") == "" {
|
||||
log.Warn("the settings changed in a way this process applies only when it starts; "+
|
||||
"it is not run by systemd, so start it again by hand", "why", reason)
|
||||
return
|
||||
}
|
||||
// The leader last, so its standbys are already on the new
|
||||
// settings when it hands over to one of them.
|
||||
delay := time.Duration(0)
|
||||
if admin != nil && admin.Leadership() != nil && admin.Leadership().IsLeader() && rt.Store() != nil {
|
||||
if cps, err := rt.Store().ControlPlanes(ctx); err == nil && len(cps) > 1 {
|
||||
delay = 30 * time.Second
|
||||
}
|
||||
}
|
||||
log.Warn("starting again to apply the settings", "why", reason, "in", delay)
|
||||
restartReason.Store(&reason)
|
||||
time.AfterFunc(delay, cancelRun)
|
||||
},
|
||||
}
|
||||
go watch.run(ctx)
|
||||
|
||||
g, gctx := errgroup.WithContext(ctx)
|
||||
g.Go(func() error { return serve(apiSrv, apiLn) })
|
||||
g.Go(func() error { return serve(adminSrv, adminLn) })
|
||||
@@ -617,8 +726,19 @@ func newServerCmd() *cobra.Command {
|
||||
log.Info("shutting down")
|
||||
sctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
// The lease first: a standby polls for it, and every agent
|
||||
// about to be let go will be refused by a standby that has
|
||||
// not taken it yet.
|
||||
if admin != nil && admin.Leadership() != nil {
|
||||
admin.Leadership().StepDown()
|
||||
}
|
||||
if agentSrv != nil {
|
||||
_ = agentSrv.Shutdown(sctx)
|
||||
// Closed, not shut down gracefully: an agent's stream
|
||||
// never goes idle, so Shutdown would wait out its whole
|
||||
// timeout while every node stayed on a control plane
|
||||
// that is leaving, and a standby could not be found
|
||||
// until it gave up. Closed, they move at once.
|
||||
_ = agentSrv.Close()
|
||||
}
|
||||
if metricsSrv != nil {
|
||||
_ = metricsSrv.Shutdown(sctx)
|
||||
@@ -626,7 +746,12 @@ func newServerCmd() *cobra.Command {
|
||||
_ = adminSrv.Shutdown(sctx)
|
||||
return apiSrv.Shutdown(sctx)
|
||||
})
|
||||
return g.Wait()
|
||||
err = g.Wait()
|
||||
if r := restartReason.Load(); r != nil {
|
||||
// Not a failure, but said as one, so systemd starts it again.
|
||||
return fmt.Errorf("stopped to start again on changed settings (%s)", *r)
|
||||
}
|
||||
return err
|
||||
},
|
||||
}
|
||||
|
||||
@@ -724,6 +849,14 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
|
||||
// Domains held for maintenance and labels set on nodes.
|
||||
ctl.Facility().UseStore(ctx, st)
|
||||
// A standby read these when it started; the leader before it may
|
||||
// have changed them since.
|
||||
lead.OnLead(func() {
|
||||
if ch := ctl.EngineChoices(); ch != nil {
|
||||
ch.Reload(ctx)
|
||||
}
|
||||
ctl.Facility().Reload(ctx)
|
||||
})
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
|
||||
@@ -121,6 +121,7 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
||||
{"connect", sa.connect},
|
||||
{"binary", sa.binary},
|
||||
{"directories", sa.dirs},
|
||||
{"this certificate", sa.ownCertificate},
|
||||
{"certificate", sa.certificate},
|
||||
{"configuration", sa.configuration},
|
||||
{"store", sa.storeAccess},
|
||||
@@ -132,6 +133,10 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
||||
fn func(context.Context) error
|
||||
}{"firewall", sa.firewall})
|
||||
}
|
||||
steps = append(steps, struct {
|
||||
name string
|
||||
fn func(context.Context) error
|
||||
}{"mirror", sa.mirror})
|
||||
if sa.vip != "" {
|
||||
steps = append(steps, struct {
|
||||
name string
|
||||
@@ -150,10 +155,18 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
||||
// firewall keeps the new control plane's internal ports to the networks it
|
||||
// reaches this one on. Not fatal: it serves without one.
|
||||
func (sa *serverAdd) firewall(ctx context.Context) error {
|
||||
cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
|
||||
cfg, err := loadEffectiveConfig(filepath.Join(sa.etcDir, "server.yaml"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// What applies the cluster's firewall setting there when it changes.
|
||||
if err := installFirewallSync(ctx, sa.remote); err != nil {
|
||||
fmt.Fprintf(sa.out, " !! firewall: %v\n", err)
|
||||
}
|
||||
if !cfg.Firewall.Enabled {
|
||||
sa.kept("firewall: off, as the cluster's settings say (firewall.enabled)")
|
||||
return nil
|
||||
}
|
||||
self, err := sourceIPToward(sa.ip)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -227,6 +240,40 @@ sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw`, nil)
|
||||
return err
|
||||
}
|
||||
|
||||
// ownCertificate makes this control plane's certificate one it can also
|
||||
// present as a client — a standby asks the leader for the console's data
|
||||
// with it — reissuing one from before that with the same names. The running
|
||||
// server reads it as it starts.
|
||||
func (sa *serverAdd) ownCertificate(context.Context) error {
|
||||
path := filepath.Join(sa.etcDir, "server.pem")
|
||||
cert, err := readCert(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if clientCapable(cert) {
|
||||
sa.kept("this control plane's certificate")
|
||||
return nil
|
||||
}
|
||||
ca, err := loadCA(sa.pkiDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the authority's key is not in %s: run this where it is (%v)", sa.pkiDir, err)
|
||||
}
|
||||
hosts := strings.Split(sanNames(cert), ", ")
|
||||
leaf, err := ca.IssuePeer(cert.Subject.CommonName, hosts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(path, leaf.CertPEM, 0o644); err != nil { //nolint:gosec // a certificate
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(sa.etcDir, "server-key.pem"), leaf.KeyPEM, 0o640); err != nil {
|
||||
return err
|
||||
}
|
||||
sa.done("this control plane's certificate reissued for %s, to ask the leader as well as answer;\n"+
|
||||
" restart llmgw-server here for it to take effect — %s takes over meanwhile", strings.Join(hosts, ", "), sa.name)
|
||||
return nil
|
||||
}
|
||||
|
||||
// certificate issues it a server certificate from this cluster's authority,
|
||||
// for its name and addresses, unless it holds one that already names them.
|
||||
func (sa *serverAdd) certificate(ctx context.Context) error {
|
||||
@@ -251,14 +298,14 @@ func (sa *serverAdd) certificate(ctx context.Context) error {
|
||||
for _, w := range want {
|
||||
missing = missing || !slices.Contains(names, w)
|
||||
}
|
||||
if !missing {
|
||||
if !missing && clientCapable(cert) {
|
||||
sa.kept("certificate for %s", strings.Join(names, ", "))
|
||||
return sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
leaf, err := ca.IssueServer(sa.name, want)
|
||||
leaf, err := ca.IssuePeer(sa.name, want)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -419,6 +466,38 @@ func (sa *serverAdd) service(ctx context.Context) error {
|
||||
}
|
||||
}
|
||||
|
||||
// mirror checks that the mirror's management API answers the new control
|
||||
// plane. It answers only the addresses and certificates it was told to, and
|
||||
// one told only of the first control plane refuses the second the moment it
|
||||
// leads: models cannot be fetched, updated or served from the registry.
|
||||
//
|
||||
// Said, not fixed: the mirror is its own machine with its own unit, and
|
||||
// llmgw does not log in to it.
|
||||
func (sa *serverAdd) mirror(ctx context.Context) error {
|
||||
cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
|
||||
if err != nil || cfg.Mirror.API == "" {
|
||||
return nil
|
||||
}
|
||||
if cfg.Mirror.TLS.Enabled() {
|
||||
fmt.Fprintf(sa.out, " note the mirror at %s answers only the certificates it names: add\n"+
|
||||
" --api-client %s to its llmgw-mirror unit, or it refuses %s when it leads\n",
|
||||
cfg.Mirror.API, sa.name, sa.name)
|
||||
return nil
|
||||
}
|
||||
code, _ := sa.sh(ctx, "curl -s -o /dev/null -m 3 -w '%{http_code}' "+shellQuote(strings.TrimRight(cfg.Mirror.API, "/")+"/")+" || true", nil)
|
||||
switch code = strings.TrimSpace(code); code {
|
||||
case "403":
|
||||
fmt.Fprintf(sa.out, " !! the mirror at %s refuses %s: add --api-allow %s to its\n"+
|
||||
" llmgw-mirror unit and restart it, or models cannot be fetched while %s leads\n",
|
||||
cfg.Mirror.API, sa.name, sa.ip, sa.name)
|
||||
case "", "000":
|
||||
fmt.Fprintf(sa.out, " !! the mirror at %s does not answer %s\n", cfg.Mirror.API, sa.name)
|
||||
default:
|
||||
sa.done("the mirror at %s answers it", cfg.Mirror.API)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// floating keeps an address on whichever of the two leads, with keepalived
|
||||
// on both, following /leader.
|
||||
func (sa *serverAdd) floating(ctx context.Context) error {
|
||||
@@ -453,6 +532,12 @@ sudo systemctl restart keepalived`, []byte(conf)); err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
}
|
||||
// Where the summary reads it, on both: readable, unlike keepalived's.
|
||||
for _, r := range []onboard.Remote{here, sa.remote} {
|
||||
if out, err := r.Run(ctx, "sudo sh -c 'cat > "+floatingPath+" && chmod 0644 "+floatingPath+"'", []byte(sa.vip+"\n")); err != nil {
|
||||
return fmt.Errorf("recording it: %v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
}
|
||||
sa.done("%s follows whichever control plane leads (keepalived on both)", sa.vip)
|
||||
return nil
|
||||
}
|
||||
@@ -503,3 +588,9 @@ vrrp_instance llmgw {
|
||||
}
|
||||
`, iface, priority, pass, vip, prefix)
|
||||
}
|
||||
|
||||
// clientCapable says a control plane's certificate can also present itself
|
||||
// as a client, as a standby does to the leader.
|
||||
func clientCapable(cert *x509.Certificate) bool {
|
||||
return slices.Contains(cert.ExtKeyUsage, x509.ExtKeyUsageClientAuth)
|
||||
}
|
||||
@@ -288,7 +288,7 @@ func (in *serverInit) judgeWanted() bool {
|
||||
// configuration written before init installed one is never changed, so the
|
||||
// model is not fetched for nothing.
|
||||
func (in *serverInit) configJudgesWith(model string) bool {
|
||||
cfg, err := config.Load(in.configPath())
|
||||
cfg, err := loadEffectiveConfig(in.configPath())
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
@@ -462,7 +462,10 @@ func (in *serverInit) ensureCerts(context.Context) error {
|
||||
// A certificate from this authority naming every address is left
|
||||
// alone. One naming fewer is reissued with the union, so an address
|
||||
// that worked yesterday is never dropped by adding one today.
|
||||
if len(missing) == 0 && cert.CheckSignatureFrom(ca.Cert) == nil {
|
||||
// And one that can also present itself as a client, which a
|
||||
// standby's admin API needs to reach the leader's; certificates
|
||||
// from before that are reissued.
|
||||
if len(missing) == 0 && cert.CheckSignatureFrom(ca.Cert) == nil && clientCapable(cert) {
|
||||
in.kept("server certificate for %s", strings.Join(have, ", "))
|
||||
return nil
|
||||
}
|
||||
@@ -477,7 +480,7 @@ func (in *serverInit) ensureCerts(context.Context) error {
|
||||
if name == "" {
|
||||
name = "llmgw-control"
|
||||
}
|
||||
leaf, err := ca.IssueServer(name, want)
|
||||
leaf, err := ca.IssuePeer(name, want)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -818,3 +821,58 @@ Before people outside this network use it:
|
||||
# in server.yaml, and copy them off this machine
|
||||
`)
|
||||
}
|
||||
|
||||
// primaryAddr is the address other machines most likely reach this one on:
|
||||
// the source of its default route, else the first address on an interface
|
||||
// that is not a container's or a bridge's. The first address of all is not
|
||||
// it on a machine with Docker, whose 172.17.0.1 a node takes for its own.
|
||||
func primaryAddr() string {
|
||||
if out, err := exec.Command("ip", "-4", "route", "get", "1.1.1.1").Output(); err == nil {
|
||||
if src := routeSource(string(out)); src != "" {
|
||||
return src
|
||||
}
|
||||
}
|
||||
ifaces, err := net.Interfaces()
|
||||
if err == nil {
|
||||
for _, ifc := range ifaces {
|
||||
if ifc.Flags&net.FlagUp == 0 || virtualInterface(ifc.Name) {
|
||||
continue
|
||||
}
|
||||
addrs, _ := ifc.Addrs()
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.To4() != nil && n.IP.IsGlobalUnicast() {
|
||||
return n.IP.String()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if ips := localAddrs(); len(ips) > 0 {
|
||||
return ips[0].String()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// routeSource reads the src of `ip route get`.
|
||||
func routeSource(out string) string {
|
||||
f := strings.Fields(out)
|
||||
for i := 0; i+1 < len(f); i++ {
|
||||
if f[i] == "src" {
|
||||
if ip := net.ParseIP(f[i+1]); ip != nil && !ip.IsLoopback() {
|
||||
return ip.String()
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// virtualInterface is an interface containers, bridges or overlays made,
|
||||
// whose address no other machine reaches this one on.
|
||||
func virtualInterface(name string) bool {
|
||||
for _, p := range []string{"docker", "br-", "virbr", "veth", "cni", "flannel", "cali", "vxlan",
|
||||
"kube", "lxc", "lxd", "podman", "cilium", "weave", "tun", "tap"} {
|
||||
if strings.HasPrefix(name, p) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bundle"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
@@ -60,7 +61,7 @@ func (in *serverInit) offline() bool {
|
||||
if in.bundle != "" {
|
||||
return true
|
||||
}
|
||||
cfg, err := config.Load(in.configPath())
|
||||
cfg, err := loadEffectiveConfig(in.configPath())
|
||||
return err == nil && cfg.Offline.Enabled()
|
||||
}
|
||||
|
||||
@@ -76,7 +77,7 @@ func (in *serverInit) mirrorRoot() string {
|
||||
// server.yaml that the cluster is offline. With a mirror on another
|
||||
// machine, the files are imported there; here only the checksums are kept.
|
||||
func (in *serverInit) ensureBundle(ctx context.Context) error {
|
||||
cfg, err := config.Load(in.configPath())
|
||||
cfg, err := loadEffectiveConfig(in.configPath())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -129,6 +130,24 @@ func (in *serverInit) ensureBundle(ctx context.Context) error {
|
||||
if sums == config.DefaultOfflineChecksums {
|
||||
sums = ""
|
||||
}
|
||||
// In the cluster's settings once the store holds them; in server.yaml
|
||||
// before, where the first start brings it in.
|
||||
stored, err := changeStoredSettings(in.configPath(), "offline, from server init --bundle", func(doc *settings.Doc) error {
|
||||
if err := doc.SetYAML([]string{"offline", "mirror"}, files); err != nil {
|
||||
return err
|
||||
}
|
||||
if sums != "" {
|
||||
return doc.SetYAML([]string{"offline", "checksums"}, sums)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if stored {
|
||||
in.done("the cluster's settings: offline, nodes install from the mirror at %s", files)
|
||||
return nil
|
||||
}
|
||||
if err := appendOfflineBlock(in.configPath(), files, sums); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -86,8 +86,8 @@ func (in *serverInit) reachableAddr() string {
|
||||
if len(in.hosts) > 0 {
|
||||
return in.hosts[0]
|
||||
}
|
||||
if ips := localAddrs(); len(ips) > 0 {
|
||||
return ips[0].String()
|
||||
if a := primaryAddr(); a != "" {
|
||||
return a
|
||||
}
|
||||
name, _ := os.Hostname()
|
||||
return name
|
||||
|
||||
@@ -19,10 +19,19 @@ import (
|
||||
// the engines and etcd when they are here. The client API is not among
|
||||
// them; it is meant to be reached.
|
||||
func (in *serverInit) ensureFirewall(ctx context.Context) error {
|
||||
cfg, err := config.Load(in.configPath())
|
||||
cfg, err := loadEffectiveConfig(in.configPath())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// What applies the cluster's setting here when it changes: the server
|
||||
// writes it, as llmgw; this applies it, as root.
|
||||
if err := in.ensureFirewallSync(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if !cfg.Firewall.Enabled {
|
||||
in.kept("firewall: off, as the cluster's settings say (firewall.enabled)")
|
||||
return nil
|
||||
}
|
||||
ports := controlPlanePorts(cfg)
|
||||
if !in.noMirror && in.mirrorURL == "" {
|
||||
ports = append(ports, firewall.MirrorFiles)
|
||||
@@ -122,3 +131,9 @@ func loopbackListen(listen string) bool {
|
||||
ip := net.ParseIP(host)
|
||||
return host == "localhost" || (ip != nil && ip.IsLoopback())
|
||||
}
|
||||
|
||||
// ensureFirewallSync installs the units that apply the cluster's firewall
|
||||
// setting on this control plane.
|
||||
func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
|
||||
return installFirewallSync(ctx, firewall.Local{})
|
||||
}
|
||||
@@ -172,3 +172,22 @@ func TestInitKeepsCertificatesFromAnAuthorityElsewhere(t *testing.T) {
|
||||
t.Error("--host cannot be honoured without the authority, and should say so")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheAddressNodesReachIsNotAContainersBridge(t *testing.T) {
|
||||
if got := routeSource("1.1.1.1 via 10.104.130.129 dev ens192 src 10.104.130.140 uid 0\n cache\n"); got != "10.104.130.140" {
|
||||
t.Errorf("route source %q", got)
|
||||
}
|
||||
if got := routeSource("unreachable"); got != "" {
|
||||
t.Errorf("no route: %q", got)
|
||||
}
|
||||
for _, name := range []string{"docker0", "br-3f2a", "virbr0", "veth12ab", "cni0", "flannel.1"} {
|
||||
if !virtualInterface(name) {
|
||||
t.Errorf("%s taken for a real interface", name)
|
||||
}
|
||||
}
|
||||
for _, name := range []string{"eth0", "ens192", "enp3s0", "bond0", "ib0"} {
|
||||
if virtualInterface(name) {
|
||||
t.Errorf("%s taken for a virtual one", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -95,6 +95,39 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
|
||||
fmt.Fprintf(out, "\nMODELS %d of %d have the replicas they asked for\n",
|
||||
s.Models.Satisfied, s.Models.Total)
|
||||
|
||||
// The control planes, when there are several: who leads, whether the
|
||||
// others would be there to take over, and the address clients use.
|
||||
if cps := s.ControlPlanes; len(cps.Members) > 0 {
|
||||
var parts []string
|
||||
leader := false
|
||||
for _, cp := range cps.Members {
|
||||
p := cp.Name
|
||||
switch cp.State {
|
||||
case "leading":
|
||||
leader = true
|
||||
p += " leads"
|
||||
default:
|
||||
p += " " + cp.State
|
||||
}
|
||||
if cp.This {
|
||||
p += " (this one)"
|
||||
}
|
||||
parts = append(parts, p)
|
||||
}
|
||||
if !leader {
|
||||
parts = append([]string{"NO LEADER — nothing is placed"}, parts...)
|
||||
}
|
||||
line := strings.Join(parts, ", ")
|
||||
if cps.Floating != "" {
|
||||
if cps.FloatingUp {
|
||||
line += " · " + cps.Floating + " on the leader"
|
||||
} else {
|
||||
line += " · " + cps.Floating + " NOT ANSWERING"
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(out, "CONTROL %s\n", line)
|
||||
}
|
||||
|
||||
// The store, when it is a replicated cluster: one line, saying who leads
|
||||
// and whether every other member keeps up.
|
||||
switch {
|
||||
|
||||
+224
-38
@@ -4,8 +4,10 @@ import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
@@ -42,6 +44,7 @@ const (
|
||||
cReplicas
|
||||
cUsage
|
||||
cQuotas
|
||||
cGuard
|
||||
cFederation
|
||||
cMirror
|
||||
cJobs
|
||||
@@ -53,18 +56,20 @@ const (
|
||||
cManual
|
||||
cSettings
|
||||
cAddNode
|
||||
cAddControl
|
||||
cReplicate
|
||||
cServe
|
||||
cUpgrade
|
||||
cRestartAll
|
||||
cBackup
|
||||
cWipe
|
||||
cLeave
|
||||
cCount
|
||||
)
|
||||
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Replicate the store", "Serve a model", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Leave"}
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
|
||||
|
||||
// consoleAction is what is done once the screen closes: each runs its own
|
||||
// screen, and the console comes back after.
|
||||
@@ -135,15 +140,18 @@ type consoleModel struct {
|
||||
node bool
|
||||
|
||||
// The settings.
|
||||
file string
|
||||
doc *settings.Doc
|
||||
cfg any // the file as loaded, for the values that apply
|
||||
levels []settingsLevel
|
||||
docErr error
|
||||
saveNote string
|
||||
file string
|
||||
doc *settings.Doc
|
||||
cfg any // the file as loaded, for the values that apply
|
||||
// settingsBase is the version of the cluster's settings the screen
|
||||
// read, when the cluster keeps them (doc is then layered over them).
|
||||
settingsBase int64
|
||||
levels []settingsLevel
|
||||
docErr error
|
||||
saveNote string
|
||||
|
||||
// A question open as a dialog.
|
||||
dialog string // "", "target", "edit", "addname", "saved", "confirm-leave", "error"
|
||||
dialog string // "", "target", "edit", "addname", "wipe", "saved", "note", "confirm-leave", "error"
|
||||
dialogTitle string
|
||||
dialogText string
|
||||
prompt textField
|
||||
@@ -167,6 +175,7 @@ type consoleModel struct {
|
||||
|
||||
quotas []adminapi.QuotaView
|
||||
quotasErr error
|
||||
guard guardState
|
||||
|
||||
st *storeHolder
|
||||
people peopleData
|
||||
@@ -215,6 +224,26 @@ func (m *consoleModel) loadSettings() {
|
||||
} else if !m.node {
|
||||
m.cfg = config.Default()
|
||||
}
|
||||
// The cluster's keys from the cluster's settings, when the control
|
||||
// plane answers with them; its own keys from its file.
|
||||
if !m.node && m.client != nil {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
var v adminapi.SettingsView
|
||||
if err := m.client.get(ctx, "/api/v1/settings", &v); err == nil && v.Version > 0 {
|
||||
if cdoc, err := settings.FromBytes("the cluster's settings", []byte(v.Body)); err == nil {
|
||||
m.doc.Layer(cdoc, config.IsLocal)
|
||||
m.settingsBase = v.Version
|
||||
if file, err := config.ReadFile(m.file); err == nil {
|
||||
if cluster, err := config.Parse([]byte(v.Body)); err == nil {
|
||||
if cfg, err := config.FromMap(config.Layer(file, cluster)); err == nil {
|
||||
m.cfg = cfg
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if root, err := settings.Root(m.file); err == nil && len(m.levels) == 0 {
|
||||
m.levels = []settingsLevel{{title: filepathBase(m.file), block: root, kind: settings.Block}}
|
||||
}
|
||||
@@ -290,6 +319,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
if m.menu == cQuotas {
|
||||
cmds = append(cmds, m.fetchQuotas())
|
||||
}
|
||||
if m.menu == cGuard {
|
||||
cmds = append(cmds, m.fetchGuard())
|
||||
}
|
||||
if m.menu == cMirror {
|
||||
cmds = append(cmds, m.fetchMirror())
|
||||
}
|
||||
@@ -348,6 +380,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
m.people = peopleData(msg)
|
||||
case consoleQuotasMsg:
|
||||
m.quotas, m.quotasErr = msg.rows, msg.err
|
||||
case consoleGuardMsg:
|
||||
window := m.guard.window
|
||||
m.guard = guardState{view: msg.view, events: msg.events, err: msg.err, window: window}
|
||||
case evalSuitesMsg:
|
||||
return m.onEvalSuites(msg)
|
||||
case evalPollMsg:
|
||||
@@ -373,7 +408,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
|
||||
func (m consoleModel) entryOK(i int) bool {
|
||||
if m.node {
|
||||
return i == cSettings || i == cManual || i == cLeave
|
||||
return i == cSettings || i == cManual || i == cWipe || i == cLeave
|
||||
}
|
||||
return i != cReplicate || !m.storeReplicated()
|
||||
}
|
||||
@@ -447,10 +482,20 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
switch m.menu {
|
||||
case cAddNode:
|
||||
return m.ask("Add a node", "user@address of the machine", 1)
|
||||
case cAddControl:
|
||||
if !m.storeReplicated() {
|
||||
m.dialog, m.dialogTitle = "error", "Add a control plane"
|
||||
m.dialogText = "A second control plane needs a store both can reach, and this one's\n" +
|
||||
"is on one machine. Replicate the store first (Do → Replicate the store)."
|
||||
return m, nil
|
||||
}
|
||||
return m.ask("Add a control plane", "user@address of the machine", 2)
|
||||
case cReplicate:
|
||||
return m.ask("Replicate the store", "user@address of database machine", 2)
|
||||
case cLeave:
|
||||
return m.leave()
|
||||
case cWipe:
|
||||
return m.askWipe()
|
||||
case cUpgrade, cRestartAll, cBackup:
|
||||
return m, m.doEntry()
|
||||
case cOverview:
|
||||
@@ -491,6 +536,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.menuD = m.replicaMenu(m.data.replicas[m.row])
|
||||
case cQuotas:
|
||||
m.menuD = m.quotaMenu(m.quotas[m.row])
|
||||
case cGuard:
|
||||
m.menuD = m.guardAttachmentMenu(m.guard.view.Attachments[m.row])
|
||||
case cMirror:
|
||||
m.menuD = m.mirrorMenu(m.mirrorGroups()[m.row])
|
||||
case cJobs:
|
||||
@@ -546,6 +593,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
switch m.menu {
|
||||
case cQuotas:
|
||||
m.quotaForm(adminapi.QuotaView{})
|
||||
case cGuard:
|
||||
m.guardAttachForm()
|
||||
case cModels:
|
||||
m.addModelForm()
|
||||
case cMirror:
|
||||
@@ -557,6 +606,11 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
case cOrgs:
|
||||
m.createOrgForm()
|
||||
}
|
||||
case "w":
|
||||
if m.menu == cGuard {
|
||||
m.guard.window = (m.guard.window + 1) % len(guardWindows)
|
||||
return m, m.fetchGuard()
|
||||
}
|
||||
case "i":
|
||||
if m.menu == cMirror {
|
||||
m.mirrorImportOllamaForm()
|
||||
@@ -594,6 +648,8 @@ func (m consoleModel) onEnter() tea.Cmd {
|
||||
return m.fetchUsage()
|
||||
case cQuotas:
|
||||
return m.fetchQuotas()
|
||||
case cGuard:
|
||||
return m.fetchGuard()
|
||||
case cMirror:
|
||||
return m.fetchMirror()
|
||||
case cMonitor:
|
||||
@@ -614,10 +670,10 @@ func (m consoleModel) peopleView() bool {
|
||||
}
|
||||
|
||||
func (m consoleModel) leave() (tea.Model, tea.Cmd) {
|
||||
if m.doc != nil && m.doc.Changed > 0 {
|
||||
if m.doc != nil && m.doc.Pending() > 0 {
|
||||
m.dialog, m.dialogTitle = "confirm-leave", "Unsaved changes"
|
||||
m.dialogText = fmt.Sprintf("%d change(s) to %s are not saved.\nLeave without saving them?\n\n%s",
|
||||
m.doc.Changed, m.file, sDim.Render("Y leave · N stay"))
|
||||
m.dialogText = fmt.Sprintf("%d change(s) are not saved.\nLeave without saving them?\n\n%s",
|
||||
m.doc.Pending(), sDim.Render("Y leave · N stay"))
|
||||
return m, nil
|
||||
}
|
||||
return m, tea.Quit
|
||||
@@ -634,7 +690,7 @@ func (m consoleModel) ask(title, question string, n int) (tea.Model, tea.Cmd) {
|
||||
|
||||
func (m consoleModel) dialogKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
switch m.dialog {
|
||||
case "error":
|
||||
case "error", "note":
|
||||
m.dialog = ""
|
||||
return m, nil
|
||||
case "confirm-leave":
|
||||
@@ -659,19 +715,45 @@ func (m consoleModel) dialogKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
v := strings.TrimSpace(m.prompt.Value())
|
||||
switch m.dialog {
|
||||
case "target":
|
||||
if v == "" {
|
||||
// The floating address is the one answer that may be left empty.
|
||||
floating := m.menu == cAddControl && len(m.asked) == 1
|
||||
if v == "" && !floating {
|
||||
return m, nil
|
||||
}
|
||||
if floating && v != "" {
|
||||
if _, _, err := net.ParseCIDR(v); err != nil {
|
||||
m.dialogText = floatingQuestion + "\n\n" + sBad.Render("not an address with its prefix, like 10.0.0.161/24")
|
||||
return m, nil
|
||||
}
|
||||
}
|
||||
m.asked = append(m.asked, v)
|
||||
m.prompt.SetValue("")
|
||||
if len(m.asked) == int(m.editKind) {
|
||||
kind := "node"
|
||||
if m.menu == cReplicate {
|
||||
kind = "patroni"
|
||||
switch m.menu {
|
||||
case cAddControl:
|
||||
args := []string{"server", "add", m.asked[0]}
|
||||
if m.asked[1] != "" {
|
||||
args = append(args, "--vip", m.asked[1])
|
||||
}
|
||||
m.action = consoleAction{kind: "run", targets: args}
|
||||
case cReplicate:
|
||||
m.action = consoleAction{kind: "patroni", targets: m.asked}
|
||||
default:
|
||||
m.action = consoleAction{kind: "node", targets: m.asked}
|
||||
}
|
||||
m.action = consoleAction{kind: kind, targets: m.asked}
|
||||
return m, tea.Quit
|
||||
}
|
||||
if m.menu == cAddControl {
|
||||
m.dialogText = floatingQuestion
|
||||
m.prompt.Placeholder = "10.0.0.161/24"
|
||||
}
|
||||
case "wipe":
|
||||
if host, _ := os.Hostname(); v != host {
|
||||
m.dialogText = m.wipeQuestion() + "\n\n" + sBad.Render("that is not this machine's name; nothing is removed")
|
||||
return m, nil
|
||||
}
|
||||
m.action = consoleAction{kind: "wipe"}
|
||||
return m, tea.Quit
|
||||
case "edit":
|
||||
if err := m.doc.Set(m.editPath, m.editKind, m.prompt.Value()); err != nil {
|
||||
m.dialogText = m.editQuestion() + "\n\n" + sBad.Render(err.Error())
|
||||
@@ -869,28 +951,60 @@ func (m consoleModel) settingsKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
}
|
||||
|
||||
func (m consoleModel) save() (tea.Model, tea.Cmd) {
|
||||
if m.doc.Changed == 0 {
|
||||
if m.doc.Pending() == 0 {
|
||||
m.dialog, m.dialogTitle, m.dialogText = "error", "Nothing to save", "No change has been made."
|
||||
return m, nil
|
||||
}
|
||||
backup, err := m.doc.Save(func(p string) error { _, err := m.load(p); return err })
|
||||
if err != nil {
|
||||
notSaved := func(err error) (tea.Model, tea.Cmd) {
|
||||
m.dialog, m.dialogTitle = "error", "Not saved"
|
||||
m.dialogText = "The configuration would not load:\n\n" + sBad.Render(err.Error()) + "\n\n" + sDim.Render("Nothing was written. Any key to go back.")
|
||||
m.dialogText = "The configuration would not load:\n\n" + sBad.Render(err.Error()) + "\n\n" +
|
||||
sDim.Render("Nothing was written. Any key to go back.")
|
||||
return m, nil
|
||||
}
|
||||
var said []string
|
||||
// The cluster's settings first: checked by the control plane against
|
||||
// its own file, and in force everywhere once kept.
|
||||
if c := m.doc.Cluster(); c != nil && c.Changed > 0 {
|
||||
body, err := c.Bytes()
|
||||
if err != nil {
|
||||
return notSaved(err)
|
||||
}
|
||||
req, _ := json.Marshal(adminapi.SettingsRequest{Base: m.settingsBase, Body: string(body), Note: "console"})
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
var v adminapi.SettingsView
|
||||
if err := m.client.put(ctx, "/api/v1/settings", req, &v); err != nil {
|
||||
return notSaved(err)
|
||||
}
|
||||
c.Changed = 0
|
||||
said = append(said, fmt.Sprintf("The cluster's settings are version %d: in force on every control plane "+
|
||||
"within seconds.\nllmgw ctl config history lists the versions; rollback goes back to one.", v.Version))
|
||||
}
|
||||
fileSaved := m.doc.Changed > 0
|
||||
if fileSaved {
|
||||
backup, err := m.doc.Save(func(p string) error { _, err := m.load(p); return err })
|
||||
if err != nil {
|
||||
return notSaved(err)
|
||||
}
|
||||
line := "Saved " + m.file + "."
|
||||
if backup != "" {
|
||||
line += " The old file is " + backup + "."
|
||||
}
|
||||
said = append(said, line)
|
||||
}
|
||||
m.loadSettings()
|
||||
note := ""
|
||||
if backup != "" {
|
||||
note = "The old file is " + backup + ".\n"
|
||||
if !m.node {
|
||||
if fileSaved {
|
||||
said = append(said, "llmgw-server notices the change and applies it within seconds.")
|
||||
}
|
||||
m.dialog, m.dialogTitle = "note", "Saved"
|
||||
m.dialogText = strings.Join(said, "\n\n") + "\n\n" + sDim.Render("Any key to go back.")
|
||||
return m, nil
|
||||
}
|
||||
m.dialog, m.dialogTitle = "saved", "Saved"
|
||||
m.dialogText = "Saved " + m.file + ".\n" + note +
|
||||
fmt.Sprintf("\n%s reads it when it starts. Restart it now?\n", m.service())
|
||||
if !m.node {
|
||||
m.dialogText += sDim.Render("Another control plane keeps its own copy: make the change there too.") + "\n"
|
||||
}
|
||||
m.dialogText += "\n" + sDim.Render("Y restart · N later")
|
||||
m.dialogText = strings.Join(said, "\n") + "\n" +
|
||||
fmt.Sprintf("\n%s reads it when it starts. Restart it now?\n", m.service()) +
|
||||
"\n" + sDim.Render("Y restart · N later")
|
||||
return m, nil
|
||||
}
|
||||
|
||||
@@ -910,6 +1024,8 @@ func (m consoleModel) rowCount() int {
|
||||
return len(m.usage)
|
||||
case cQuotas:
|
||||
return len(m.quotas)
|
||||
case cGuard:
|
||||
return len(m.guard.view.Attachments)
|
||||
case cMirror:
|
||||
return len(m.mirrorGroups())
|
||||
case cJobs:
|
||||
@@ -949,7 +1065,7 @@ func (m consoleModel) View() string {
|
||||
nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Help", Heading: true})
|
||||
case cSettings:
|
||||
nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Settings", Heading: true})
|
||||
case cAddNode, cLeave:
|
||||
case cAddNode, cWipe:
|
||||
if (i == cAddNode) != m.node {
|
||||
nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Do", Heading: true})
|
||||
}
|
||||
@@ -961,8 +1077,8 @@ func (m consoleModel) View() string {
|
||||
if n := runningJobs(m.data.jobs); i == cJobs && n > 0 {
|
||||
label += fmt.Sprintf(" (%d)", n)
|
||||
}
|
||||
if i == cSettings && m.doc != nil && m.doc.Changed > 0 {
|
||||
label += fmt.Sprintf(" (%d)", m.doc.Changed)
|
||||
if i == cSettings && m.doc != nil && m.doc.Pending() > 0 {
|
||||
label += fmt.Sprintf(" (%d)", m.doc.Pending())
|
||||
}
|
||||
nav = append(nav, navItem{Label: label, Current: i == m.menu, Dim: !m.entryOK(i)})
|
||||
}
|
||||
@@ -990,7 +1106,7 @@ func (m consoleModel) View() string {
|
||||
f.DialogTitle = m.dialogTitle
|
||||
f.Dialog = m.dialogText
|
||||
switch m.dialog {
|
||||
case "target", "edit", "addname":
|
||||
case "target", "edit", "addname", "wipe":
|
||||
q := m.dialogText
|
||||
if m.dialog == "target" && m.editKind > 1 {
|
||||
q = fmt.Sprintf("%s %d of %d", q, len(m.asked)+1, int(m.editKind))
|
||||
@@ -1036,7 +1152,7 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
c := &cobra.Command{}
|
||||
c.SetOut(&b)
|
||||
printSummary(c, m.data.sum)
|
||||
return "Overview", "\n" + indent(b.String(), " "), keys
|
||||
return "Overview", m.standbyNote() + "\n" + indent(b.String(), " "), keys
|
||||
case cNodes, cLogs:
|
||||
if m.detail && m.row < len(m.data.nodes) {
|
||||
n := m.data.nodes[m.row]
|
||||
@@ -1137,6 +1253,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
return "Usage", head + table([]string{"WHO", "REQUESTS", "TOKENS", "ERRORS", "SPENT"}, rows, m.row, pw, m.inPanel), keys
|
||||
case cQuotas:
|
||||
return m.quotasPanel(pw)
|
||||
case cGuard:
|
||||
return m.guardPanel(pw)
|
||||
case cAccounts, cKeys, cOrgs:
|
||||
return m.peoplePanel(pw)
|
||||
case cManual:
|
||||
@@ -1173,6 +1291,10 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
case cAddNode:
|
||||
item = "\n Adds a machine as a node: installs llmgw on it over ssh, checks it,\n gives it a certificate and starts its agent.\n\n " +
|
||||
sDim.Render("On the command line:") + "\n sudo llmgw node add <user>@<address>"
|
||||
case cAddControl:
|
||||
item = "\n Makes another machine a control plane of this cluster: a standby\n that takes over within seconds when this one is lost. Every node is\n told of it; none is touched. With a floating address, keepalived on\n both keeps it on whichever leads — the address to give clients.\n\n " +
|
||||
sDim.Render("Needs the store replicated first, so both reach it.") + "\n\n " +
|
||||
sDim.Render("On the command line:") + "\n sudo --preserve-env=SSH_AUTH_SOCK llmgw server add <user>@<address> --vip <address>/<prefix>"
|
||||
case cReplicate:
|
||||
item = "\n Moves the store onto two machines under Patroni, replicated,\n so losing one loses nothing.\n\n " +
|
||||
sDim.Render("On the command line:") +
|
||||
@@ -1186,6 +1308,9 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
case cBackup:
|
||||
item = "\n Writes the store and /etc/llmgw to one archive in " + DefaultBackupDir + ",\n and keeps the last few. Copy it off this machine.\n\n " +
|
||||
sDim.Render("On the command line:") + "\n sudo llmgw server backup"
|
||||
case cWipe:
|
||||
item = "\n Removes llmgw from this machine: its services, firewall table,\n configuration, certificates, the authority, secrets, state and users,\n so it can be set up again from nothing. Model weights, engines, the\n store's database and backups stay. Asks for the machine's name first.\n\n " +
|
||||
sDim.Render("On the command line, with --models, --store or --all for more:") + "\n sudo llmgw wipe"
|
||||
case cLeave:
|
||||
item = "\n Closes the console. The cluster keeps running."
|
||||
}
|
||||
@@ -1509,6 +1634,16 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
|
||||
// Already set up: only the store changes.
|
||||
in.noMirror, in.noNode, in.noModel = true, true, true
|
||||
err = in.runInitTUI(ctx, out)
|
||||
case "wipe":
|
||||
c := Root()
|
||||
c.SetArgs([]string{"wipe", "--yes"})
|
||||
c.SetOut(out)
|
||||
c.SetErr(out)
|
||||
if err := c.ExecuteContext(ctx); err != nil {
|
||||
fmt.Fprintln(out, sBad.Render("✗ ")+err.Error())
|
||||
}
|
||||
// Nothing is left here for the console to show.
|
||||
return nil
|
||||
case "run":
|
||||
c := Root()
|
||||
c.SetArgs(act.targets)
|
||||
@@ -1531,3 +1666,54 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
|
||||
_, _ = bufio.NewReader(os.Stdin).ReadString('\n')
|
||||
}
|
||||
}
|
||||
|
||||
// floatingQuestion is the second question when adding a control plane.
|
||||
const floatingQuestion = "A floating address for clients, kept on whichever control plane leads,\n" +
|
||||
"as 10.0.0.161/24 — a free one on this network. Empty for none, when a load\n" +
|
||||
"balancer in front checks /ready instead."
|
||||
|
||||
// standbyNote says, on a control plane standing by, whose cluster the
|
||||
// console shows: the leader's, which this one's admin API passes it on to.
|
||||
func (m consoleModel) standbyNote() string {
|
||||
var here, leader string
|
||||
for _, cp := range m.data.sum.ControlPlanes.Members {
|
||||
if cp.This {
|
||||
here = cp.State
|
||||
}
|
||||
if cp.State == "leading" {
|
||||
leader = cp.Name
|
||||
}
|
||||
}
|
||||
if here != "standing by" || leader == "" {
|
||||
return ""
|
||||
}
|
||||
return "\n " + sDim.Render("Standing by. Shown, and changed, on the leader: "+leader) + "\n"
|
||||
}
|
||||
|
||||
// askWipe asks for this machine's name before removing llmgw from it, saying
|
||||
// what would go.
|
||||
func (m consoleModel) askWipe() (tea.Model, tea.Cmd) {
|
||||
if os.Geteuid() != 0 {
|
||||
m.dialog, m.dialogTitle = "error", "Wipe this machine"
|
||||
m.dialogText = "Removing services, users and files under /etc needs root:\nstart the console with sudo llmgw."
|
||||
return m, nil
|
||||
}
|
||||
m.dialog, m.dialogTitle, m.dialogText = "wipe", "Wipe this machine", m.wipeQuestion()
|
||||
m.prompt = newTextField()
|
||||
m.prompt.Width = 40
|
||||
m.prompt.Focus()
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (m consoleModel) wipeQuestion() string {
|
||||
host, _ := os.Hostname()
|
||||
w := &wiper{}
|
||||
var b strings.Builder
|
||||
b.WriteString("This removes, on " + host + ":\n\n")
|
||||
for _, s := range w.plan() {
|
||||
b.WriteString(" - " + s.what + "\n")
|
||||
}
|
||||
b.WriteString("\nModel weights, engines, the store's database and backups stay\n" +
|
||||
"(sudo llmgw wipe --all takes them too).\n\nType the machine's name to remove it all:")
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// The guardrails, as the console shows them: the policies server.yaml
|
||||
// defines and what each checks, where they are attached, whether the judge
|
||||
// models they lean on are serving, and what they did in a window.
|
||||
|
||||
// guardWindows are the windows W steps through.
|
||||
var guardWindows = []string{"24h", "7d", "1h"}
|
||||
|
||||
type guardState struct {
|
||||
view adminapi.GuardView
|
||||
events adminapi.GuardEventsView
|
||||
err error
|
||||
// window is an index into guardWindows.
|
||||
window int
|
||||
}
|
||||
|
||||
type consoleGuardMsg struct {
|
||||
view adminapi.GuardView
|
||||
events adminapi.GuardEventsView
|
||||
err error
|
||||
}
|
||||
|
||||
func (m consoleModel) fetchGuard() tea.Cmd {
|
||||
c, since := m.client, guardWindows[m.guard.window]
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
var msg consoleGuardMsg
|
||||
if msg.err = c.get(ctx, "/api/v1/guard", &msg.view); msg.err != nil {
|
||||
return msg
|
||||
}
|
||||
// What they did is a second question; a store without the record
|
||||
// still shows the policies.
|
||||
_ = c.get(ctx, "/api/v1/guard/events?since="+since, &msg.events)
|
||||
return msg
|
||||
}
|
||||
}
|
||||
|
||||
func (m consoleModel) guardPanel(pw int) (string, string, string) {
|
||||
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||
if m.inPanel {
|
||||
keys = "↑↓ attachment · Enter actions · A attach · W window · ← back · Q leave"
|
||||
}
|
||||
g := m.guard
|
||||
var b strings.Builder
|
||||
b.WriteString("\n " + sDim.Render("What requests and answers are checked for, where, and what that did. A to attach a policy.") + "\n\n")
|
||||
if g.err != nil {
|
||||
if strings.Contains(g.err.Error(), "501") || strings.Contains(g.err.Error(), "no guard policies") {
|
||||
b.WriteString(" " + sDim.Render("No policies: define them under guard: in server.yaml (Settings), then restart."))
|
||||
} else {
|
||||
b.WriteString(" " + sBad.Render(g.err.Error()))
|
||||
}
|
||||
return "Guardrails", b.String(), keys
|
||||
}
|
||||
if len(g.view.Details) == 0 && len(g.view.Policies) == 0 {
|
||||
b.WriteString(" " + sDim.Render("asking the control plane…"))
|
||||
return "Guardrails", b.String(), keys
|
||||
}
|
||||
|
||||
def := "none"
|
||||
if len(g.view.Default) > 0 {
|
||||
def = strings.Join(g.view.Default, ", ")
|
||||
}
|
||||
b.WriteString(" " + sTitle.Render("POLICIES") + " " + sDim.Render("on every request: "+def) + "\n")
|
||||
for _, p := range g.view.Details {
|
||||
line := fmt.Sprintf(" %-14s %s · fail %s", p.Name, p.Mode, p.Fail)
|
||||
if p.NoStore {
|
||||
line += " · keeps nothing"
|
||||
}
|
||||
if p.Judge != "" {
|
||||
line += " · judge " + p.Judge + " " + m.judgeState(p.Judge)
|
||||
}
|
||||
b.WriteString(line + "\n")
|
||||
for _, c := range p.Checks {
|
||||
b.WriteString(" " + strings.Repeat(" ", 15) + sDim.Render(c) + "\n")
|
||||
}
|
||||
}
|
||||
|
||||
b.WriteString("\n " + sTitle.Render("ATTACHED") + " " + sDim.Render("besides the default, to part of the tenancy tree, a person or a key") + "\n")
|
||||
if len(g.view.Attachments) == 0 {
|
||||
b.WriteString(" " + sDim.Render("Nowhere. A to attach one.") + "\n")
|
||||
} else {
|
||||
var rows [][]string
|
||||
for _, a := range g.view.Attachments {
|
||||
who := a.Name
|
||||
if who == "" {
|
||||
who = a.ScopeID
|
||||
}
|
||||
rows = append(rows, []string{a.Scope, who, a.Policy})
|
||||
}
|
||||
b.WriteString(table([]string{"SCOPE", "WHO", "POLICY"}, rows, m.row, pw, m.inPanel) + "\n")
|
||||
}
|
||||
|
||||
window := guardWindows[g.window]
|
||||
b.WriteString("\n " + sTitle.Render("IN THE LAST "+strings.ToUpper(window)) + " " + sDim.Render("W for another window") + "\n")
|
||||
if len(g.events.Counts) == 0 {
|
||||
b.WriteString(" " + sDim.Render("Nothing flagged, redacted or blocked.") + "\n")
|
||||
} else {
|
||||
var rows [][]string
|
||||
for i, c := range g.events.Counts {
|
||||
if i == 10 {
|
||||
break
|
||||
}
|
||||
rows = append(rows, []string{c.Action, c.Policy, dash(c.Category), dash(c.Project), fmt.Sprint(c.Count)})
|
||||
}
|
||||
b.WriteString(table([]string{"ACTION", "POLICY", "CATEGORY", "PROJECT", "REQUESTS"}, rows, -1, pw, false) + "\n")
|
||||
}
|
||||
if len(g.events.Events) > 0 {
|
||||
b.WriteString("\n " + sTitle.Render("NEWEST") + "\n")
|
||||
var rows [][]string
|
||||
for i, e := range g.events.Events {
|
||||
if i == 10 {
|
||||
break
|
||||
}
|
||||
rows = append(rows, []string{e.At.Local().Format("01-02 15:04:05"), e.Action, e.Policy, dash(e.Category),
|
||||
e.Model, fmt.Sprint(e.Status), fmt.Sprintf("%dms", e.GuardMs), e.RequestID})
|
||||
}
|
||||
b.WriteString(table([]string{"AT", "ACTION", "POLICY", "CATEGORY", "MODEL", "STATUS", "GUARD", "REQUEST"}, rows, -1, pw, false))
|
||||
}
|
||||
return "Guardrails", b.String(), keys
|
||||
}
|
||||
|
||||
// judgeState is how many of a judge model's replicas are ready: a policy
|
||||
// that fails closed refuses everything while it has none.
|
||||
func (m consoleModel) judgeState(model string) string {
|
||||
for _, md := range m.data.models {
|
||||
if md.Name == model {
|
||||
s := fmt.Sprintf("(%d/%d ready)", md.Ready, md.Desired)
|
||||
if md.Ready == 0 {
|
||||
return sBad.Render(s)
|
||||
}
|
||||
return sOK.Render(s)
|
||||
}
|
||||
}
|
||||
return sBad.Render("(not in the catalog)")
|
||||
}
|
||||
|
||||
// guardAttachForm attaches a policy.
|
||||
func (m *consoleModel) guardAttachForm() {
|
||||
if len(m.guard.view.Policies) == 0 {
|
||||
return
|
||||
}
|
||||
m.openForm(&formDialog{title: "Attach a guardrail policy",
|
||||
note: "Requests from there are checked by it as well as by the default.",
|
||||
fields: []formField{
|
||||
choiceF("to", []string{"project", "team", "org", "user", "key"}, "project"),
|
||||
textF("which", "", "ORG/TEAM/PROJECT, ORG/TEAM, ORG, or the ID"),
|
||||
choiceF("policy", m.guard.view.Policies, m.guard.view.Policies[0]),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
if strings.TrimSpace(v[1]) == "" {
|
||||
return nil, fmt.Errorf("which %s?", v[0])
|
||||
}
|
||||
req := adminapi.GuardAttachmentRequest{Target: v[0] + ":" + strings.TrimSpace(v[1]), Policy: v[2]}
|
||||
return m.call("Attach "+v[2], http.MethodPost, "/api/v1/guard/attachments", req, nil), nil
|
||||
}})
|
||||
}
|
||||
|
||||
// guardAttachmentMenu is what can be done to one attachment.
|
||||
func (m consoleModel) guardAttachmentMenu(a adminapi.GuardAttachmentView) *menuDialog {
|
||||
who := a.Name
|
||||
if who == "" {
|
||||
who = a.ScopeID
|
||||
}
|
||||
return &menuDialog{title: a.Policy + " on " + a.Scope + " " + who, items: []menuItem{
|
||||
{"Detach…", func(m *consoleModel) tea.Cmd {
|
||||
m.confirmD = &confirmDialog{title: "Detach " + a.Policy,
|
||||
text: "Requests from " + who + " are then checked by the default policies alone.",
|
||||
yes: func(m *consoleModel) tea.Cmd {
|
||||
req := adminapi.GuardAttachmentRequest{Target: a.Scope + ":" + a.ScopeID, Policy: a.Policy}
|
||||
return m.call("Detach "+a.Policy, http.MethodDelete, "/api/v1/guard/attachments", req, nil)
|
||||
}}
|
||||
return nil
|
||||
}},
|
||||
}}
|
||||
}
|
||||
@@ -13,7 +13,6 @@ import (
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
@@ -57,7 +56,7 @@ var hfClients sync.Map
|
||||
|
||||
func (m consoleModel) registry() *hf.Client {
|
||||
endpoint := ""
|
||||
if cfg, err := config.Load(m.file); err == nil {
|
||||
if cfg, err := loadEffectiveConfig(m.file); err == nil {
|
||||
endpoint = cfg.Mirror.Registry
|
||||
}
|
||||
c, _ := hfClients.LoadOrStore(endpoint, hf.New(endpoint, 20*time.Second))
|
||||
|
||||
@@ -12,7 +12,6 @@ import (
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
@@ -118,7 +117,7 @@ func (m consoleModel) fetchPeople() tea.Cmd {
|
||||
}
|
||||
}
|
||||
}
|
||||
if cfg, err := config.Load(file); err == nil {
|
||||
if cfg, err := loadEffectiveConfig(file); err == nil {
|
||||
for name := range cfg.Guard.Policies {
|
||||
d.guardPolicies = append(d.guardPolicies, name)
|
||||
}
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -163,7 +167,7 @@ func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
|
||||
cm = mm.(consoleModel)
|
||||
cm.prompt.SetValue("5")
|
||||
mm = press(cm, "enter", "s")
|
||||
if d := mm.(consoleModel).dialog; d != "saved" {
|
||||
if d := mm.(consoleModel).dialog; d != "note" {
|
||||
t.Fatalf("after saving the dialog is %q: %s", d, mm.(consoleModel).dialogText)
|
||||
}
|
||||
b, _ := os.ReadFile(file)
|
||||
@@ -171,3 +175,152 @@ func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
|
||||
t.Errorf("server.yaml:\n%s", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleAddsAControlPlaneWithAFloatingAddress(t *testing.T) {
|
||||
replicated := consoleModel{menu: cAddControl, loaded: true, width: 100, height: 30}
|
||||
replicated.data.sum.Store = []adminapi.StoreMember{{Name: "a", Role: "leader"}}
|
||||
|
||||
// The address, then the floating address; the console closes to run it.
|
||||
var mm tea.Model = replicated
|
||||
mm = press(mm, "enter", "ubuntu@10.0.0.150", "enter", "10.0.0.161/24", "enter")
|
||||
want := []string{"server", "add", "ubuntu@10.0.0.150", "--vip", "10.0.0.161/24"}
|
||||
if a := mm.(consoleModel).action; a.kind != "run" || !slices.Equal(a.targets, want) {
|
||||
t.Errorf("%+v", a)
|
||||
}
|
||||
|
||||
// No floating address is an answer too.
|
||||
mm = press(replicated, "enter", "ubuntu@10.0.0.150", "enter", "enter")
|
||||
if a := mm.(consoleModel).action; !slices.Equal(a.targets, want[:3]) {
|
||||
t.Errorf("without one: %+v", a)
|
||||
}
|
||||
|
||||
// A mistyped one is caught here rather than half way through.
|
||||
mm = press(replicated, "enter", "ubuntu@10.0.0.150", "enter", "10.0.0.161", "enter")
|
||||
if c := mm.(consoleModel); c.action.kind != "" || !strings.Contains(c.dialogText, "prefix") {
|
||||
t.Errorf("an address without its prefix went through: %+v", c.action)
|
||||
}
|
||||
|
||||
// On a store on one machine it says what to do first.
|
||||
mm = press(consoleModel{menu: cAddControl, loaded: true, width: 100, height: 30}, "enter")
|
||||
if c := mm.(consoleModel); c.dialog != "error" || !strings.Contains(c.dialogText, "Replicate the store") {
|
||||
t.Errorf("dialog %q: %s", c.dialog, c.dialogText)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleOnAStandbySaysWhoseClusterItShows(t *testing.T) {
|
||||
m := consoleModel{menu: cOverview, loaded: true, width: 120, height: 40}
|
||||
m.data.sum.ControlPlanes = adminapi.ControlPlanes{Members: []adminapi.ControlPlane{
|
||||
{Name: "llmgw-ctl", State: "leading"},
|
||||
{Name: "llmgw-ctl02", State: "standing by", This: true},
|
||||
}}
|
||||
if v := m.View(); !strings.Contains(v, "Standing by. Shown, and changed, on the leader: llmgw-ctl") {
|
||||
t.Errorf("the overview on a standby:\n%s", v)
|
||||
}
|
||||
m.data.sum.ControlPlanes.Members[0].This, m.data.sum.ControlPlanes.Members[1].This = true, false
|
||||
if v := m.View(); strings.Contains(v, "Standing by.") {
|
||||
t.Error("the leader's console says it is standing by")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleShowsTheGuardrails(t *testing.T) {
|
||||
m := consoleModel{menu: cGuard, loaded: true, width: 140, height: 60}
|
||||
m.data.models = []adminapi.ModelView{{Name: "llama-guard-3-1b", Ready: 2, Desired: 2}}
|
||||
m.guard.view = adminapi.GuardView{
|
||||
Policies: []string{"baseline"}, Default: []string{"baseline"},
|
||||
Details: []adminapi.GuardPolicyView{{Name: "baseline", Mode: "enforce", Fail: "closed", Judge: "llama-guard-3-1b",
|
||||
Checks: []string{"input: pii email → redact"}}},
|
||||
Attachments: []adminapi.GuardAttachmentView{{Name: "acme/ml"}},
|
||||
}
|
||||
m.guard.view.Attachments[0].Scope, m.guard.view.Attachments[0].Policy = "team", "baseline"
|
||||
m.guard.events = adminapi.GuardEventsView{Counts: []adminapi.GuardCountView{{Action: "block", Policy: "baseline", Category: "S1", Count: 7}}}
|
||||
v := m.View()
|
||||
for _, want := range []string{"Guardrails", "baseline", "fail closed", "judge llama-guard-3-1b (2/2 ready)",
|
||||
"input: pii email → redact", "acme/ml", "IN THE LAST 24H", "S1"} {
|
||||
if !strings.Contains(v, want) {
|
||||
t.Errorf("%q missing:\n%s", want, v)
|
||||
}
|
||||
}
|
||||
// W moves the window, and asks again.
|
||||
m.inPanel = true
|
||||
mm, cmd := m.Update(tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("w")})
|
||||
if c := mm.(consoleModel); guardWindows[c.guard.window] != "7d" || cmd == nil {
|
||||
t.Errorf("window %q", guardWindows[c.guard.window])
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleEditsTheClustersSettingsInTheStore(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, "server.yaml")
|
||||
if err := os.WriteFile(file, []byte("listen: \":8080\"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var put adminapi.SettingsRequest
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == "GET" && r.URL.Path == "/api/v1/settings":
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7, Body: "router:\n retries: 2\n"})
|
||||
case r.Method == "PUT" && r.URL.Path == "/api/v1/settings":
|
||||
_ = json.NewDecoder(r.Body).Decode(&put)
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8, Body: put.Body})
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
m := newConsoleModel(strings.TrimPrefix(srv.URL, "http://"), file)
|
||||
m.width, m.height, m.loaded = 120, 40, true
|
||||
m.menu, m.inPanel = cSettings, true
|
||||
m.loadSettings()
|
||||
for i, r := range m.settingsRows() {
|
||||
if r.label == "router" {
|
||||
m.row = i
|
||||
}
|
||||
}
|
||||
var mm tea.Model = press(m, "enter")
|
||||
cm := mm.(consoleModel)
|
||||
for i, r := range cm.settingsRows() {
|
||||
if r.label == "retries" {
|
||||
cm.row = i
|
||||
}
|
||||
}
|
||||
if v := cm.View(); !strings.Contains(v, "2") {
|
||||
t.Errorf("the cluster's value is not shown:\n%s", v)
|
||||
}
|
||||
mm = press(cm, "enter")
|
||||
cm = mm.(consoleModel)
|
||||
cm.prompt.SetValue("5")
|
||||
mm = press(cm, "enter", "s")
|
||||
if d := mm.(consoleModel).dialog; d != "note" || !strings.Contains(mm.(consoleModel).dialogText, "version 8") {
|
||||
t.Fatalf("after saving: %q %s", d, mm.(consoleModel).dialogText)
|
||||
}
|
||||
if put.Base != 7 || !strings.Contains(put.Body, "retries: 5") {
|
||||
t.Errorf("sent %+v", put)
|
||||
}
|
||||
if b, _ := os.ReadFile(file); string(b) != "listen: \":8080\"\n" {
|
||||
t.Errorf("the cluster's key went into the machine's file:\n%s", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleWipesOnlyOnTheMachinesName(t *testing.T) {
|
||||
m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "server.yaml"))
|
||||
m.width, m.height, m.loaded = 120, 40, true
|
||||
m.menu = cWipe
|
||||
m.dialog, m.dialogTitle, m.dialogText = "wipe", "Wipe this machine", m.wipeQuestion()
|
||||
m.prompt = newTextField()
|
||||
m.prompt.Focus()
|
||||
if v := m.View(); !strings.Contains(v, "Type the machine's name") {
|
||||
t.Errorf("the dialog does not ask for the name:\n%s", v)
|
||||
}
|
||||
m.prompt.SetValue("not-this-one")
|
||||
mm := press(m, "enter")
|
||||
if a := mm.(consoleModel).action; a.kind != "" {
|
||||
t.Fatalf("a wrong name wiped: %+v", a)
|
||||
}
|
||||
cm := mm.(consoleModel)
|
||||
host, _ := os.Hostname()
|
||||
cm.prompt.SetValue(host)
|
||||
mm = press(cm, "enter")
|
||||
if a := mm.(consoleModel).action; a.kind != "wipe" {
|
||||
t.Errorf("the right name did not: %+v", a)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,277 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// wipeUnits are the systemd units llmgw installs, stopped first so nothing
|
||||
// writes while the rest goes.
|
||||
var wipeUnits = []string{
|
||||
"llmgw-server.service", "llmgw-agent.service", "llmgw-mirror.service",
|
||||
firewallSyncPathUnit, "llmgw-firewall-sync.service", "llmgw-firewall.service",
|
||||
"llmgw-backup.timer", "llmgw-backup.service",
|
||||
}
|
||||
|
||||
// wiper removes llmgw from a machine. root prefixes every path, for tests;
|
||||
// run runs a command.
|
||||
type wiper struct {
|
||||
root string
|
||||
run func(name string, args ...string) error
|
||||
store bool
|
||||
models bool
|
||||
all bool
|
||||
}
|
||||
|
||||
// wipeStep is one thing to remove, said before it is done.
|
||||
type wipeStep struct {
|
||||
what string
|
||||
do func() error
|
||||
}
|
||||
|
||||
func (w *wiper) path(p string) string { return filepath.Join(w.root, p) }
|
||||
|
||||
func (w *wiper) exists(p string) bool {
|
||||
_, err := os.Lstat(w.path(p))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func (w *wiper) glob(pattern string) []string {
|
||||
m, _ := filepath.Glob(w.path(pattern))
|
||||
out := make([]string, len(m))
|
||||
for i, p := range m {
|
||||
out[i] = strings.TrimPrefix(p, w.root)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// mentions is whether a file is there and says llmgw: a file of a name
|
||||
// another tool also uses (keepalived's, etcd's) is removed only when llmgw
|
||||
// wrote it.
|
||||
func (w *wiper) mentions(p string) bool {
|
||||
b, err := os.ReadFile(w.path(p))
|
||||
return err == nil && strings.Contains(string(b), "llmgw")
|
||||
}
|
||||
|
||||
func (w *wiper) remove(paths ...string) func() error {
|
||||
return func() error {
|
||||
var errs []error
|
||||
for _, p := range paths {
|
||||
if err := os.RemoveAll(w.path(p)); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
}
|
||||
|
||||
// plan is every step, in order, for what is on this machine.
|
||||
func (w *wiper) plan() []wipeStep {
|
||||
var steps []wipeStep
|
||||
add := func(what string, do func() error) { steps = append(steps, wipeStep{what, do}) }
|
||||
files := func(what string, paths ...string) {
|
||||
var there []string
|
||||
for _, p := range paths {
|
||||
if w.exists(p) {
|
||||
there = append(there, p)
|
||||
}
|
||||
}
|
||||
if len(there) > 0 {
|
||||
add(what+": "+strings.Join(there, ", "), w.remove(there...))
|
||||
}
|
||||
}
|
||||
|
||||
// The services, stopped and gone.
|
||||
var units []string
|
||||
for _, u := range wipeUnits {
|
||||
if w.exists("/etc/systemd/system/" + u) {
|
||||
units = append(units, u)
|
||||
}
|
||||
}
|
||||
if len(units) > 0 {
|
||||
add("stop and remove "+strings.Join(units, ", "), func() error {
|
||||
for _, u := range units {
|
||||
_ = w.run("systemctl", "disable", "--now", u)
|
||||
if err := os.Remove(w.path("/etc/systemd/system/" + u)); err != nil && !os.IsNotExist(err) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return w.run("systemctl", "daemon-reload")
|
||||
})
|
||||
}
|
||||
// Engines run as their own user, kept alive across the agent's restarts:
|
||||
// stopped by user, or they would keep their GPUs.
|
||||
if w.userExists("llmgw-engine") {
|
||||
add("stop the engines (every process of llmgw-engine)", func() error {
|
||||
_ = w.run("pkill", "-KILL", "-u", "llmgw-engine")
|
||||
return nil
|
||||
})
|
||||
}
|
||||
add("take llmgw's table out of the firewall (nft inet llmgw), if loaded", func() error {
|
||||
_ = w.run("nft", "delete", "table", "inet", "llmgw")
|
||||
return nil
|
||||
})
|
||||
if w.mentions("/etc/keepalived/keepalived.conf") {
|
||||
add("stop keepalived and remove llmgw's floating address (/etc/keepalived/keepalived.conf)", func() error {
|
||||
_ = w.run("systemctl", "disable", "--now", "keepalived")
|
||||
return w.remove("/etc/keepalived/keepalived.conf")()
|
||||
})
|
||||
}
|
||||
|
||||
files("configuration, certificates, the authority and secrets", "/etc/llmgw")
|
||||
files("state: the admin token, a SQLite store, what the agent kept",
|
||||
"/var/lib/llmgw", "/var/lib/llmgw-agent")
|
||||
files("the sudo grant setup left", "/etc/sudoers.d/llmgw-setup")
|
||||
if w.exists("/usr/local/share/ca-certificates/llmgw-ca.crt") {
|
||||
add("the cluster's authority from the system's trust (llmgw-ca.crt)", func() error {
|
||||
if err := w.remove("/usr/local/share/ca-certificates/llmgw-ca.crt")(); err != nil {
|
||||
return err
|
||||
}
|
||||
_ = w.run("update-ca-certificates", "--fresh")
|
||||
return nil
|
||||
})
|
||||
}
|
||||
if w.exists("/etc/pki/ca-trust/source/anchors/llmgw-ca.pem") {
|
||||
add("the cluster's authority from the system's trust (llmgw-ca.pem)", func() error {
|
||||
if err := w.remove("/etc/pki/ca-trust/source/anchors/llmgw-ca.pem")(); err != nil {
|
||||
return err
|
||||
}
|
||||
_ = w.run("update-ca-trust")
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
if w.store || w.all {
|
||||
w.planStore(add, files)
|
||||
}
|
||||
if w.models || w.all {
|
||||
files("model weights and the mirror", "/srv/models", "/var/lib/llmgw-weights", "/var/lib/llmgw-models")
|
||||
files("installed engines", "/opt/llmgw")
|
||||
}
|
||||
if w.all {
|
||||
files("backups", DefaultBackupDir)
|
||||
}
|
||||
|
||||
for _, u := range []string{"llmgw-engine", "llmgw"} {
|
||||
if w.userExists(u) {
|
||||
add("the user "+u, func() error { return w.run("userdel", u) })
|
||||
}
|
||||
}
|
||||
if w.all {
|
||||
files("the llmgw binary", "/usr/local/bin/llmgw", "/usr/local/bin/llmgw.prev")
|
||||
}
|
||||
return steps
|
||||
}
|
||||
|
||||
// planStore is the store, when it is on this machine: a member of the
|
||||
// replicated one (Patroni, etcd), or llmgw's database in a local PostgreSQL.
|
||||
func (w *wiper) planStore(add func(string, func() error), files func(string, ...string)) {
|
||||
patroni := w.mentions("/etc/patroni/config.yml")
|
||||
if patroni || w.mentions("/etc/default/etcd") {
|
||||
add("stop Patroni and etcd", func() error {
|
||||
_ = w.run("systemctl", "disable", "--now", "patroni")
|
||||
_ = w.run("systemctl", "disable", "--now", "etcd")
|
||||
return nil
|
||||
})
|
||||
}
|
||||
if patroni {
|
||||
data := append(w.glob("/var/lib/postgresql/*/llmgw"), w.glob("/var/lib/pgsql/*/llmgw")...)
|
||||
files("the store's database on this member, and Patroni's configuration",
|
||||
append([]string{"/etc/patroni", "/var/lib/postgresql/.pgpass_patroni", "/root/llmgw-pg"}, data...)...)
|
||||
} else if w.exists("/usr/bin/psql") {
|
||||
add("drop the llmgw database and role from the local PostgreSQL, if there", func() error {
|
||||
_ = w.run("runuser", "-u", "postgres", "--", "psql", "-q",
|
||||
"-c", "DROP DATABASE IF EXISTS llmgw", "-c", "DROP ROLE IF EXISTS llmgw")
|
||||
return nil
|
||||
})
|
||||
}
|
||||
if w.mentions("/etc/default/etcd") {
|
||||
files("etcd's data and configuration", "/var/lib/etcd/default", "/etc/etcd/tls", "/etc/default/etcd")
|
||||
}
|
||||
}
|
||||
|
||||
func (w *wiper) userExists(name string) bool {
|
||||
b, err := os.ReadFile(w.path("/etc/passwd"))
|
||||
return err == nil && strings.Contains("\n"+string(b), "\n"+name+":")
|
||||
}
|
||||
|
||||
func newWipeCmd() *cobra.Command {
|
||||
var w wiper
|
||||
var yes bool
|
||||
c := &cobra.Command{
|
||||
Use: "wipe",
|
||||
Short: "Remove llmgw from this machine: its services, configuration and state",
|
||||
Long: "Stops and removes llmgw's services, takes its firewall table out, and\n" +
|
||||
"deletes its configuration, certificates, the authority, secrets, state\n" +
|
||||
"and users — what server init, node add, server add and mirror init left —\n" +
|
||||
"so the machine can be set up again from nothing. It says everything it\n" +
|
||||
"will remove first, and asks for the machine's name.\n\n" +
|
||||
"Kept unless asked: model weights, the mirror and engines (--models), the\n" +
|
||||
"store's database when it is on this machine (--store), backups and the\n" +
|
||||
"binary (--all, which is all of these). Packages llmgw installed —\n" +
|
||||
"PostgreSQL, Patroni, etcd, nftables, the node exporter — stay.\n\n" +
|
||||
"A node wiped is then forgotten on the control plane with\n" +
|
||||
"'llmgw node remove <name>'.",
|
||||
Example: " sudo llmgw wipe\n" +
|
||||
" sudo llmgw wipe --models\n" +
|
||||
" sudo llmgw wipe --all --yes",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if runtime.GOOS != "linux" {
|
||||
return fmt.Errorf("wipe removes llmgw from a Linux machine; this is %s", runtime.GOOS)
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return errors.New("wipe removes services, users and files under /etc: run it with sudo")
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
w.run = func(name string, args ...string) error {
|
||||
return exec.CommandContext(cmd.Context(), name, args...).Run() //nolint:gosec // fixed commands
|
||||
}
|
||||
steps := w.plan()
|
||||
host, _ := os.Hostname()
|
||||
fmt.Fprintf(out, "On %s, this removes:\n", host)
|
||||
for _, s := range steps {
|
||||
fmt.Fprintf(out, " - %s\n", s.what)
|
||||
}
|
||||
if !yes {
|
||||
if !isTerminal(out) {
|
||||
return errors.New("nothing removed: give --yes to wipe without being asked")
|
||||
}
|
||||
fmt.Fprintf(out, "\nType this machine's name (%s) to remove all of it: ", host)
|
||||
answer, _ := bufio.NewReader(cmd.InOrStdin()).ReadString('\n')
|
||||
if strings.TrimSpace(answer) != host {
|
||||
return errors.New("nothing removed")
|
||||
}
|
||||
}
|
||||
var failed []string
|
||||
for _, s := range steps {
|
||||
if err := s.do(); err != nil {
|
||||
failed = append(failed, s.what+": "+err.Error())
|
||||
fmt.Fprintf(out, " !! %s: %v\n", s.what, err)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(out, " ok %s\n", s.what)
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("%d step(s) failed; running wipe again finishes them", len(failed))
|
||||
}
|
||||
fmt.Fprintln(out, "\nllmgw is gone from this machine. If it was a node, forget it on the control plane: "+
|
||||
"llmgw node remove "+host)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := c.Flags()
|
||||
f.BoolVar(&yes, "yes", false, "do not ask")
|
||||
f.BoolVar(&w.models, "models", false, "also remove model weights, the mirror's files and installed engines")
|
||||
f.BoolVar(&w.store, "store", false, "also remove the store's database here: a Patroni member's data, etcd's, or llmgw's database in a local PostgreSQL")
|
||||
f.BoolVar(&w.all, "all", false, "everything: --models, --store, backups and the binary")
|
||||
return c
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func wipeFixture(t *testing.T) string {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
for p, body := range map[string]string{
|
||||
"/etc/llmgw/server.yaml": "listen: :8080\n",
|
||||
"/etc/llmgw/pki/ca-key.pem": "key",
|
||||
"/var/lib/llmgw/admin-token": "t",
|
||||
"/var/lib/llmgw-agent/control-planes.json": "[]",
|
||||
"/etc/systemd/system/llmgw-server.service": "[Unit]",
|
||||
"/etc/systemd/system/llmgw-agent.service": "[Unit]",
|
||||
"/etc/sudoers.d/llmgw-setup": "x",
|
||||
"/etc/keepalived/keepalived.conf": "# llmgw: the gateway's floating address\n",
|
||||
"/etc/passwd": "root:x:0:0::/root:/bin/sh\nllmgw:x:999:999::/var/lib/llmgw:/usr/sbin/nologin\nllmgw-engine:x:998:998::/:/usr/sbin/nologin\n",
|
||||
"/srv/models/qwen/v1/q.gguf": "w",
|
||||
"/opt/llmgw/engines/llamacpp/b1/llama-server": "e",
|
||||
"/etc/patroni/config.yml": "scope: llmgw\n",
|
||||
"/var/lib/postgresql/16/llmgw/PG_VERSION": "16",
|
||||
"/etc/default/etcd": "ETCD_INITIAL_CLUSTER_TOKEN=llmgw\n",
|
||||
"/var/lib/etcd/default/member/snap": "s",
|
||||
"/usr/local/bin/llmgw": "bin",
|
||||
} {
|
||||
full := filepath.Join(root, p)
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(full, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return root
|
||||
}
|
||||
|
||||
func TestWipeRemovesTheConfigurationAndKeepsWhatWasNotAsked(t *testing.T) {
|
||||
root := wipeFixture(t)
|
||||
var ran []string
|
||||
w := &wiper{root: root, run: func(name string, args ...string) error {
|
||||
ran = append(ran, name+" "+strings.Join(args, " "))
|
||||
return nil
|
||||
}}
|
||||
for _, s := range w.plan() {
|
||||
if err := s.do(); err != nil {
|
||||
t.Fatalf("%s: %v", s.what, err)
|
||||
}
|
||||
}
|
||||
for _, gone := range []string{"/etc/llmgw", "/var/lib/llmgw", "/var/lib/llmgw-agent",
|
||||
"/etc/systemd/system/llmgw-server.service", "/etc/sudoers.d/llmgw-setup", "/etc/keepalived/keepalived.conf"} {
|
||||
if _, err := os.Stat(filepath.Join(root, gone)); err == nil {
|
||||
t.Errorf("%s is still there", gone)
|
||||
}
|
||||
}
|
||||
for _, kept := range []string{"/srv/models/qwen/v1/q.gguf", "/opt/llmgw/engines", "/etc/patroni/config.yml",
|
||||
"/var/lib/postgresql/16/llmgw", "/var/lib/etcd/default", "/usr/local/bin/llmgw"} {
|
||||
if _, err := os.Stat(filepath.Join(root, kept)); err != nil {
|
||||
t.Errorf("%s was removed without being asked", kept)
|
||||
}
|
||||
}
|
||||
all := strings.Join(ran, "\n")
|
||||
for _, want := range []string{"systemctl disable --now llmgw-server.service", "pkill -KILL -u llmgw-engine",
|
||||
"nft delete table inet llmgw", "systemctl disable --now keepalived", "userdel llmgw-engine", "userdel llmgw"} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
if strings.Contains(all, "patroni") {
|
||||
t.Error("the store was touched without --store")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWipeAllTakesTheStoreAndTheModelsToo(t *testing.T) {
|
||||
root := wipeFixture(t)
|
||||
w := &wiper{root: root, all: true, run: func(string, ...string) error { return nil }}
|
||||
for _, s := range w.plan() {
|
||||
if err := s.do(); err != nil {
|
||||
t.Fatalf("%s: %v", s.what, err)
|
||||
}
|
||||
}
|
||||
for _, gone := range []string{"/srv/models", "/opt/llmgw", "/etc/patroni", "/var/lib/postgresql/16/llmgw",
|
||||
"/var/lib/etcd/default", "/etc/default/etcd", "/usr/local/bin/llmgw"} {
|
||||
if _, err := os.Stat(filepath.Join(root, gone)); err == nil {
|
||||
t.Errorf("%s is still there", gone)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWipeLeavesAKeepalivedItDidNotWrite(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
p := filepath.Join(root, "/etc/keepalived/keepalived.conf")
|
||||
_ = os.MkdirAll(filepath.Dir(p), 0o755)
|
||||
_ = os.WriteFile(p, []byte("vrrp_instance site_vip {}\n"), 0o600)
|
||||
w := &wiper{root: root, run: func(string, ...string) error { return nil }}
|
||||
for _, s := range w.plan() {
|
||||
_ = s.do()
|
||||
}
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
t.Error("someone else's keepalived configuration was removed")
|
||||
}
|
||||
}
|
||||
@@ -14,7 +14,6 @@ import (
|
||||
|
||||
"github.com/knadh/koanf/parsers/yaml"
|
||||
env "github.com/knadh/koanf/providers/env/v2"
|
||||
"github.com/knadh/koanf/providers/file"
|
||||
"github.com/knadh/koanf/v2"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
@@ -147,6 +146,9 @@ type Server struct {
|
||||
// Prometheus and Alertmanager in between. Empty sends nothing.
|
||||
Notify notify.Config `koanf:"notify"`
|
||||
|
||||
// Firewall turns llmgw's firewall on every machine on or off.
|
||||
Firewall FirewallSettings `koanf:"firewall"`
|
||||
|
||||
// FirewallNetworks are networks, besides those each machine reaches the
|
||||
// cluster on, that llmgw's internal ports answer: a monitoring host on
|
||||
// another subnet, nodes behind a router. Written into every machine's
|
||||
@@ -764,6 +766,7 @@ func Default() Server {
|
||||
AgentListen: ":7443",
|
||||
MaxConnections: 10_000,
|
||||
Mirror: Mirror{UpdateCheck: 6 * time.Hour},
|
||||
Firewall: FirewallSettings{Enabled: true},
|
||||
Control: Control{
|
||||
Interval: 5 * time.Second,
|
||||
BaseBackoff: 10 * time.Second,
|
||||
@@ -823,15 +826,30 @@ func Default() Server {
|
||||
// An empty path loads the defaults, which is what makes `llmgw server` with a
|
||||
// couple of --endpoint flags usable for a smoke test.
|
||||
func Load(path string) (Server, error) {
|
||||
cfg := Default()
|
||||
k := koanf.New(".")
|
||||
|
||||
var b []byte
|
||||
if path != "" {
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
return cfg, fmt.Errorf("config file: %w", err)
|
||||
return Default(), fmt.Errorf("config file: %w", err)
|
||||
}
|
||||
if err := k.Load(file.Provider(path), yaml.Parser()); err != nil {
|
||||
return cfg, fmt.Errorf("parsing %s: %w", path, err)
|
||||
var err error
|
||||
if b, err = os.ReadFile(path); err != nil { //nolint:gosec // the configuration
|
||||
return Default(), fmt.Errorf("config file: %w", err)
|
||||
}
|
||||
}
|
||||
cfg, err := load(b)
|
||||
if err != nil && path != "" && strings.HasPrefix(err.Error(), "parsing settings") {
|
||||
return cfg, fmt.Errorf("parsing %s:%s", path, strings.TrimPrefix(err.Error(), "parsing settings:"))
|
||||
}
|
||||
return cfg, err
|
||||
}
|
||||
|
||||
// load reads a YAML document of settings over the defaults.
|
||||
func load(b []byte) (Server, error) {
|
||||
cfg := Default()
|
||||
k := koanf.New(".")
|
||||
if len(b) > 0 {
|
||||
if err := k.Load(bytesProvider(b), yaml.Parser()); err != nil {
|
||||
return cfg, fmt.Errorf("parsing settings: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -32,3 +32,15 @@ func (s Server) EnginePorts() (int, int) {
|
||||
d := Default().Control.PortRange
|
||||
return d[0], d[1]
|
||||
}
|
||||
|
||||
// FirewallSettings is whether llmgw keeps a firewall on the cluster's
|
||||
// machines.
|
||||
type FirewallSettings struct {
|
||||
// Enabled keeps llmgw's internal ports — engines, node exporter, the
|
||||
// agents' stream, the mirror, the store — to the cluster's networks on
|
||||
// every machine. False removes llmgw's nftables table everywhere, for a
|
||||
// site whose own firewall does that job; the rules are kept, and true
|
||||
// puts them back. Changed in the cluster's settings, it is in force on
|
||||
// every node and control plane within seconds.
|
||||
Enabled bool `koanf:"enabled"`
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"os"
|
||||
"reflect"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"go.yaml.in/yaml/v3"
|
||||
)
|
||||
|
||||
// localKeys are the keys of server.yaml that belong to one machine: where it
|
||||
// listens, its certificates, how it reaches the store, its own secrets and
|
||||
// files. Everything else is the cluster's, kept in the store and the same on
|
||||
// every control plane (see Layer).
|
||||
var localKeys = [][]string{
|
||||
{"listen"}, {"admin_listen"}, {"metrics_listen"}, {"agent_listen"},
|
||||
{"max_connections"}, {"admin_token_file"}, {"admin_remote"}, {"debug_pprof"},
|
||||
{"agent_insecure"}, {"agent_tls"},
|
||||
{"auth", "store"}, {"threads", "store"},
|
||||
{"secrets"}, {"backup_dir"}, {"backup_copy_key"},
|
||||
{"tracing"}, {"monitoring"},
|
||||
{"mirror", "api"}, {"mirror", "tls"},
|
||||
{"endpoints"},
|
||||
}
|
||||
|
||||
// IsLocal is whether a key path is one machine's own, kept in its
|
||||
// server.yaml, rather than the cluster's.
|
||||
func IsLocal(path []string) bool {
|
||||
for _, l := range localKeys {
|
||||
if len(path) >= len(l) && slices.Equal(path[:len(l)], l) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// LocalKeys are the per-machine keys, dotted, for saying which they are.
|
||||
func LocalKeys() []string {
|
||||
out := make([]string, len(localKeys))
|
||||
for i, l := range localKeys {
|
||||
out[i] = strings.Join(l, ".")
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ReadFile is a configuration file as YAML maps, empty for no path.
|
||||
func ReadFile(path string) (map[string]any, error) {
|
||||
if path == "" {
|
||||
return map[string]any{}, nil
|
||||
}
|
||||
b, err := os.ReadFile(path) //nolint:gosec // the configuration
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("config file: %w", err)
|
||||
}
|
||||
m, err := Parse(b)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parsing %s: %w", path, err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// Parse reads a YAML document of settings.
|
||||
func Parse(b []byte) (map[string]any, error) {
|
||||
m := map[string]any{}
|
||||
if len(bytes.TrimSpace(b)) == 0 {
|
||||
return m, nil
|
||||
}
|
||||
if err := yaml.Unmarshal(b, &m); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if m == nil {
|
||||
m = map[string]any{}
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// Split separates settings into one machine's and the cluster's.
|
||||
func Split(m map[string]any) (local, cluster map[string]any) {
|
||||
local, cluster = map[string]any{}, map[string]any{}
|
||||
split(m, nil, local, cluster)
|
||||
return local, cluster
|
||||
}
|
||||
|
||||
func split(m map[string]any, path []string, local, cluster map[string]any) {
|
||||
for k, v := range m {
|
||||
p := append(slices.Clone(path), k)
|
||||
switch {
|
||||
case IsLocal(p):
|
||||
local[k] = v
|
||||
case hasLocalBelow(p):
|
||||
sub, ok := v.(map[string]any)
|
||||
if !ok {
|
||||
cluster[k] = v
|
||||
continue
|
||||
}
|
||||
l, c := map[string]any{}, map[string]any{}
|
||||
split(sub, p, l, c)
|
||||
if len(l) > 0 {
|
||||
local[k] = l
|
||||
}
|
||||
if len(c) > 0 {
|
||||
cluster[k] = c
|
||||
}
|
||||
default:
|
||||
cluster[k] = v
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// hasLocalBelow is whether some local key lies under a path: auth holds
|
||||
// store, which is local, beside its policies, which are not.
|
||||
func hasLocalBelow(path []string) bool {
|
||||
for _, l := range localKeys {
|
||||
if len(l) > len(path) && slices.Equal(l[:len(path)], path) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Layer is what a control plane runs on: its own keys from its server.yaml
|
||||
// and every other one from the cluster's settings. The file's cluster keys
|
||||
// are not read once the cluster has settings of its own; Ignored names them.
|
||||
func Layer(file, cluster map[string]any) map[string]any {
|
||||
local, _ := Split(file)
|
||||
out := clone(local)
|
||||
merge(out, cluster)
|
||||
return out
|
||||
}
|
||||
|
||||
// Ignored are the cluster keys a file has that the cluster's settings, once
|
||||
// there, take the place of.
|
||||
func Ignored(file map[string]any) []string {
|
||||
_, c := Split(file)
|
||||
return flatKeys(c, "")
|
||||
}
|
||||
|
||||
func flatKeys(m map[string]any, prefix string) []string {
|
||||
var out []string
|
||||
for _, k := range slices.Sorted(maps.Keys(m)) {
|
||||
p := prefix + k
|
||||
if sub, ok := m[k].(map[string]any); ok && hasLocalBelow(strings.Split(p, ".")) {
|
||||
out = append(out, flatKeys(sub, p+".")...)
|
||||
continue
|
||||
}
|
||||
out = append(out, p)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// CheckCluster refuses a cluster document that sets one machine's keys: a
|
||||
// listener or a certificate path written there would be every control
|
||||
// plane's, and none of them could start on it.
|
||||
func CheckCluster(m map[string]any) error {
|
||||
local, _ := Split(m)
|
||||
if keys := flatKeys(local, ""); len(keys) > 0 {
|
||||
return fmt.Errorf("%s belong to each machine's server.yaml, not to the cluster's settings",
|
||||
strings.Join(keys, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Changed are the keys whose values differ between two sets of settings,
|
||||
// at the top level, and below it where a key mixes local and cluster ones.
|
||||
func Changed(a, b map[string]any) []string {
|
||||
var out []string
|
||||
changed(a, b, nil, &out)
|
||||
slices.Sort(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func changed(a, b map[string]any, path []string, out *[]string) {
|
||||
keys := map[string]bool{}
|
||||
for k := range a {
|
||||
keys[k] = true
|
||||
}
|
||||
for k := range b {
|
||||
keys[k] = true
|
||||
}
|
||||
for k := range keys {
|
||||
p := append(slices.Clone(path), k)
|
||||
va, vb := a[k], b[k]
|
||||
if reflect.DeepEqual(va, vb) {
|
||||
continue
|
||||
}
|
||||
ma, okA := va.(map[string]any)
|
||||
mb, okB := vb.(map[string]any)
|
||||
if hasLocalBelow(p) && (okA || va == nil) && (okB || vb == nil) {
|
||||
changed(ma, mb, p, out)
|
||||
continue
|
||||
}
|
||||
*out = append(*out, strings.Join(p, "."))
|
||||
}
|
||||
}
|
||||
|
||||
// Marshal writes settings as YAML.
|
||||
func Marshal(m map[string]any) ([]byte, error) {
|
||||
if len(m) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
var b bytes.Buffer
|
||||
enc := yaml.NewEncoder(&b)
|
||||
enc.SetIndent(2)
|
||||
if err := enc.Encode(m); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return b.Bytes(), enc.Close()
|
||||
}
|
||||
|
||||
// FromMap is the configuration settings make: laid over the defaults, with
|
||||
// LLMGW_ environment overrides, and validated.
|
||||
func FromMap(m map[string]any) (Server, error) {
|
||||
b, err := Marshal(m)
|
||||
if err != nil {
|
||||
return Default(), err
|
||||
}
|
||||
return load(b)
|
||||
}
|
||||
|
||||
// bytesProvider hands koanf a document already read.
|
||||
type bytesProvider []byte
|
||||
|
||||
func (b bytesProvider) ReadBytes() ([]byte, error) { return b, nil }
|
||||
func (b bytesProvider) Read() (map[string]any, error) {
|
||||
return nil, errors.New("bytesProvider reads bytes only")
|
||||
}
|
||||
|
||||
func clone(m map[string]any) map[string]any {
|
||||
out := make(map[string]any, len(m))
|
||||
for k, v := range m {
|
||||
if sub, ok := v.(map[string]any); ok {
|
||||
v = clone(sub)
|
||||
}
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// merge lays src over dst: maps key by key, everything else replaced.
|
||||
func merge(dst, src map[string]any) {
|
||||
for k, v := range src {
|
||||
if sv, ok := v.(map[string]any); ok {
|
||||
if dv, ok := dst[k].(map[string]any); ok {
|
||||
merge(dv, sv)
|
||||
continue
|
||||
}
|
||||
v = clone(sv)
|
||||
}
|
||||
dst[k] = v
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const machineFile = `
|
||||
listen: ":8080"
|
||||
agent_listen: ":7443"
|
||||
agent_insecure: true
|
||||
auth:
|
||||
enabled: true
|
||||
store: /var/lib/llmgw/llmgw.db
|
||||
default_rpm: 60
|
||||
mirror:
|
||||
api: http://127.0.0.1:8082
|
||||
update_check: 6h
|
||||
guard:
|
||||
default: [baseline]
|
||||
policies:
|
||||
baseline:
|
||||
input:
|
||||
injection: flag
|
||||
router:
|
||||
retries: 2
|
||||
`
|
||||
|
||||
func TestAFileSplitsIntoTheMachinesAndTheClusters(t *testing.T) {
|
||||
file, err := Parse([]byte(machineFile))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
local, cluster := Split(file)
|
||||
for _, k := range []string{"listen", "agent_listen", "agent_insecure"} {
|
||||
if _, ok := local[k]; !ok {
|
||||
t.Errorf("%s is not the machine's", k)
|
||||
}
|
||||
}
|
||||
if local["auth"].(map[string]any)["store"] == nil || cluster["auth"].(map[string]any)["store"] != nil {
|
||||
t.Errorf("auth.store went to the cluster: local %v cluster %v", local["auth"], cluster["auth"])
|
||||
}
|
||||
if cluster["auth"].(map[string]any)["default_rpm"] != 60 {
|
||||
t.Errorf("auth's policy stayed with the machine: %v", cluster["auth"])
|
||||
}
|
||||
if local["mirror"].(map[string]any)["api"] == nil || cluster["mirror"].(map[string]any)["update_check"] == nil {
|
||||
t.Errorf("mirror split wrong: local %v cluster %v", local["mirror"], cluster["mirror"])
|
||||
}
|
||||
for _, k := range []string{"guard", "router"} {
|
||||
if _, ok := cluster[k]; !ok {
|
||||
t.Errorf("%s is not the cluster's", k)
|
||||
}
|
||||
}
|
||||
if err := CheckCluster(cluster); err != nil {
|
||||
t.Errorf("the cluster half is refused: %v", err)
|
||||
}
|
||||
if err := CheckCluster(file); err == nil || !strings.Contains(err.Error(), "auth.store") {
|
||||
t.Errorf("a cluster document with a machine's keys: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClustersSettingsTakeThePlaceOfTheFiles(t *testing.T) {
|
||||
file, _ := Parse([]byte(machineFile))
|
||||
cluster, _ := Parse([]byte("router:\n retries: 5\nauth:\n enabled: true\n default_rpm: 10\n"))
|
||||
eff := Layer(file, cluster)
|
||||
cfg, err := FromMap(eff)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.Router.Retries != 5 || cfg.Auth.DefaultRPM != 10 {
|
||||
t.Errorf("cluster values not in force: retries %d rpm %d", cfg.Router.Retries, cfg.Auth.DefaultRPM)
|
||||
}
|
||||
if cfg.Auth.Store != "/var/lib/llmgw/llmgw.db" || cfg.Mirror.API != "http://127.0.0.1:8082" || cfg.Listen != ":8080" {
|
||||
t.Errorf("the machine's own keys lost: %+v", cfg.Auth)
|
||||
}
|
||||
if len(cfg.Guard.Policies) != 0 {
|
||||
t.Error("the file's guard applies though the cluster has settings without one")
|
||||
}
|
||||
if got := Ignored(file); !slices.Contains(got, "guard") || !slices.Contains(got, "auth.default_rpm") ||
|
||||
slices.Contains(got, "auth.store") {
|
||||
t.Errorf("ignored: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChangedNamesWhatMoved(t *testing.T) {
|
||||
a, _ := Parse([]byte(machineFile))
|
||||
b, _ := Parse([]byte(strings.Replace(machineFile, "retries: 2", "retries: 3", 1)))
|
||||
if got := Changed(a, b); !slices.Equal(got, []string{"router"}) {
|
||||
t.Errorf("changed %v", got)
|
||||
}
|
||||
c, _ := Parse([]byte(strings.Replace(machineFile, "store: /var/lib/llmgw/llmgw.db", "store: /x.db", 1)))
|
||||
if got := Changed(a, c); !slices.Equal(got, []string{"auth.store"}) {
|
||||
t.Errorf("changed %v", got)
|
||||
}
|
||||
if got := Changed(a, a); len(got) != 0 {
|
||||
t.Errorf("unchanged: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -54,6 +54,9 @@ type API struct {
|
||||
releases string
|
||||
// storeStatus reads a replicated store's members; nil for one database.
|
||||
storeStatus func() ([]StoreMember, error)
|
||||
// controlPlanes says what each control plane answers; nil for a cluster
|
||||
// of one.
|
||||
controlPlanes func() ControlPlanes
|
||||
// serve runs registry-to-replica jobs; nil without a mirror and a store.
|
||||
serve *modelserve.Server
|
||||
// history is the Prometheus the console's charts read from.
|
||||
@@ -64,7 +67,8 @@ type API struct {
|
||||
// nil without a store.
|
||||
federation *federation.Service
|
||||
// guard is the guardrail policies' library and attachments.
|
||||
guard *GuardAdmin
|
||||
guard *GuardAdmin
|
||||
settings *Settings
|
||||
// revoke records a removed node, so its certificate is refused.
|
||||
revoke func(ctx context.Context, nodeID string, at time.Time) error
|
||||
|
||||
@@ -108,6 +112,7 @@ func (a *API) Register(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
a.registerQuotas(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
|
||||
mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)
|
||||
mux.HandleFunc("POST /api/v1/notify/test", a.notifyTest)
|
||||
@@ -870,6 +875,10 @@ func (a *API) SetUsage(u Usage) { a.usage = u }
|
||||
// SetStoreStatus supplies the state of a replicated store, for the summary.
|
||||
func (a *API) SetStoreStatus(f func() ([]StoreMember, error)) { a.storeStatus = f }
|
||||
|
||||
// SetControlPlanes sets where the summary learns which control plane leads
|
||||
// and which stand by.
|
||||
func (a *API) SetControlPlanes(f func() ControlPlanes) { a.controlPlanes = f }
|
||||
|
||||
// ScopeUsageView is one group's spend, with its quota when it has one.
|
||||
type ScopeUsageView struct {
|
||||
Scope string `json:"scope"`
|
||||
@@ -1155,8 +1164,18 @@ func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, Removed{Removed: name})
|
||||
}
|
||||
|
||||
func (a *API) summary(w http.ResponseWriter, _ *http.Request) {
|
||||
writeJSON(w, http.StatusOK, a.Summary())
|
||||
func (a *API) summary(w http.ResponseWriter, r *http.Request) {
|
||||
s := a.Summary()
|
||||
// Asked through a standby, which passes on the console's questions to
|
||||
// the leader: "this one" is the standby the console is on.
|
||||
if by := r.Header.Get("X-Llmgw-Forwarded-By"); by != "" && len(s.ControlPlanes.Members) > 0 {
|
||||
members := slices.Clone(s.ControlPlanes.Members)
|
||||
for i := range members {
|
||||
members[i].This = members[i].Name == by
|
||||
}
|
||||
s.ControlPlanes.Members = members
|
||||
}
|
||||
writeJSON(w, http.StatusOK, s)
|
||||
}
|
||||
|
||||
func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
@@ -4,11 +4,14 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// GuardStore is where policy attachments are kept.
|
||||
@@ -18,10 +21,11 @@ type GuardStore interface {
|
||||
DetachGuard(ctx context.Context, a guard.Attachment) (bool, error)
|
||||
}
|
||||
|
||||
// GuardAdmin is what the guard endpoints need: the library from server.yaml,
|
||||
// the store, and a way to have the gate read attachments again at once.
|
||||
// GuardAdmin is what the guard endpoints need: the library in force (the
|
||||
// cluster's settings), the store, and a way to have the gate read
|
||||
// attachments again at once.
|
||||
type GuardAdmin struct {
|
||||
Library guard.Config
|
||||
Library func() guard.Config
|
||||
Store GuardStore
|
||||
Invalidate func()
|
||||
}
|
||||
@@ -34,6 +38,114 @@ type GuardView struct {
|
||||
Policies []string `json:"policies"`
|
||||
Default []string `json:"default"`
|
||||
Attachments []GuardAttachmentView `json:"attachments"`
|
||||
// Details says what each policy does, in the order of Policies.
|
||||
Details []GuardPolicyView `json:"details"`
|
||||
}
|
||||
|
||||
// GuardPolicyView is what one policy does, in brief.
|
||||
type GuardPolicyView struct {
|
||||
Name string `json:"name"`
|
||||
// Mode is enforce, shadow or off; Fail is open or closed.
|
||||
Mode string `json:"mode"`
|
||||
Fail string `json:"fail"`
|
||||
NoStore bool `json:"no_store,omitempty"`
|
||||
// Judge is the judge model, when one reads the text.
|
||||
Judge string `json:"judge,omitempty"`
|
||||
// Checks are what it looks for, one line each: "input: pii email → redact".
|
||||
Checks []string `json:"checks"`
|
||||
}
|
||||
|
||||
// GuardHistory is what the guardrails did, from the usage record; the store
|
||||
// has it, and a store that does not leaves the events endpoint unanswered.
|
||||
type GuardHistory interface {
|
||||
GuardEvents(ctx context.Context, since time.Time, limit int) ([]store.GuardEvent, error)
|
||||
GuardCounts(ctx context.Context, since time.Time) ([]store.GuardCount, error)
|
||||
}
|
||||
|
||||
// GuardEventsView is what the guardrails did since a moment: totals, and the
|
||||
// newest requests.
|
||||
type GuardEventsView struct {
|
||||
Since time.Time `json:"since"`
|
||||
Counts []GuardCountView `json:"counts"`
|
||||
Events []GuardEventView `json:"events"`
|
||||
}
|
||||
|
||||
// GuardCountView is how many requests one policy did one thing to.
|
||||
type GuardCountView struct {
|
||||
Project string `json:"project,omitempty"`
|
||||
Action string `json:"action"`
|
||||
Policy string `json:"policy"`
|
||||
Category string `json:"category,omitempty"`
|
||||
Count int64 `json:"count"`
|
||||
}
|
||||
|
||||
// GuardEventView is one request the guardrails did something to.
|
||||
type GuardEventView struct {
|
||||
At time.Time `json:"at"`
|
||||
RequestID string `json:"request_id"`
|
||||
Project string `json:"project,omitempty"`
|
||||
Model string `json:"model"`
|
||||
Status int `json:"status"`
|
||||
Action string `json:"action"`
|
||||
Policy string `json:"policy"`
|
||||
Category string `json:"category,omitempty"`
|
||||
GuardMs int64 `json:"guard_ms"`
|
||||
}
|
||||
|
||||
// policyView sums one policy up.
|
||||
func policyView(name string, p guard.Policy) GuardPolicyView {
|
||||
v := GuardPolicyView{Name: name, Mode: or(p.Mode, "enforce"), Fail: or(p.Fail, "open"), NoStore: p.NoStore, Checks: []string{}}
|
||||
rules := func(dir string, r guard.Rules) {
|
||||
for _, kind := range slices.Sorted(maps.Keys(r.PII)) {
|
||||
v.Checks = append(v.Checks, fmt.Sprintf("%s: pii %s → %s", dir, kind, r.PII[kind]))
|
||||
}
|
||||
if r.Injection != "" {
|
||||
v.Checks = append(v.Checks, fmt.Sprintf("%s: prompt injection → %s", dir, r.Injection))
|
||||
}
|
||||
for _, d := range r.Deny {
|
||||
v.Checks = append(v.Checks, fmt.Sprintf("%s: pattern %s → %s", dir, d.Name, or(d.Action, "block")))
|
||||
}
|
||||
if r.MaxChars > 0 {
|
||||
v.Checks = append(v.Checks, fmt.Sprintf("%s: at most %d characters", dir, r.MaxChars))
|
||||
}
|
||||
}
|
||||
rules("input", p.Input)
|
||||
rules("output", p.Output)
|
||||
if j := p.Judge; j != nil {
|
||||
v.Judge = j.Model
|
||||
var dirs []string
|
||||
if j.Input != "" {
|
||||
dirs = append(dirs, "input → "+j.Input)
|
||||
}
|
||||
if j.Output != "" {
|
||||
dirs = append(dirs, "output → "+j.Output)
|
||||
}
|
||||
line := "judge " + j.Model + ": " + strings.Join(dirs, ", ")
|
||||
if len(j.Categories) > 0 {
|
||||
line += " (" + strings.Join(j.Categories, " ") + ")"
|
||||
}
|
||||
v.Checks = append(v.Checks, line)
|
||||
}
|
||||
if len(p.Tools.Allow) > 0 {
|
||||
v.Checks = append(v.Checks, "tools: only "+strings.Join(p.Tools.Allow, ", "))
|
||||
}
|
||||
if len(p.Tools.Deny) > 0 {
|
||||
v.Checks = append(v.Checks, "tools: never "+strings.Join(p.Tools.Deny, ", "))
|
||||
}
|
||||
if p.MaxOutputTokens > 0 {
|
||||
v.Checks = append(v.Checks, fmt.Sprintf("answers of at most %d tokens", p.MaxOutputTokens))
|
||||
}
|
||||
if p.SystemPrompt != "" {
|
||||
v.Checks = append(v.Checks, "a system prompt of its own")
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func or(s, dflt string) string {
|
||||
if s == "" {
|
||||
return dflt
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// GuardAttachmentView is one attachment, with the name people know its
|
||||
@@ -43,6 +155,15 @@ type GuardAttachmentView struct {
|
||||
Name string `json:"name,omitempty"`
|
||||
}
|
||||
|
||||
// guardHistory is where what the guardrails did is read, when there is one.
|
||||
func (a *API) guardHistory() (GuardHistory, bool) {
|
||||
if a.guard == nil {
|
||||
return nil, false
|
||||
}
|
||||
h, ok := a.guard.Store.(GuardHistory)
|
||||
return h, ok
|
||||
}
|
||||
|
||||
// guardTarget reads where a policy is to be attached.
|
||||
func (a *API) guardTarget(ctx context.Context, target string) (guard.Attachment, error) {
|
||||
kind, what, ok := strings.Cut(strings.TrimSpace(target), ":")
|
||||
@@ -79,7 +200,7 @@ type GuardAttachmentRequest struct {
|
||||
func (a *API) registerGuard(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.guard == nil {
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard: in server.yaml")
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
|
||||
return
|
||||
}
|
||||
att, err := a.guard.Store.GuardAttachments(r.Context())
|
||||
@@ -87,7 +208,7 @@ func (a *API) registerGuard(mux Mux) {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
v := GuardView{Default: a.guard.Library.Default, Attachments: []GuardAttachmentView{}}
|
||||
v := GuardView{Default: a.guard.Library().Default, Attachments: []GuardAttachmentView{}}
|
||||
var names map[string]string
|
||||
if a.quotas != nil {
|
||||
names, _ = a.scopeNames(r.Context())
|
||||
@@ -95,16 +216,67 @@ func (a *API) registerGuard(mux Mux) {
|
||||
for _, at := range att {
|
||||
v.Attachments = append(v.Attachments, GuardAttachmentView{Attachment: at, Name: names[at.Scope+":"+at.ScopeID]})
|
||||
}
|
||||
for name := range a.guard.Library.Policies {
|
||||
for name := range a.guard.Library().Policies {
|
||||
v.Policies = append(v.Policies, name)
|
||||
}
|
||||
slices.Sort(v.Policies)
|
||||
for _, name := range v.Policies {
|
||||
v.Details = append(v.Details, policyView(name, a.guard.Library().Policies[name]))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
})
|
||||
mux.HandleFunc("GET /api/v1/guard/events", func(w http.ResponseWriter, r *http.Request) {
|
||||
h, ok := a.guardHistory()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies, or no store to record what they did")
|
||||
return
|
||||
}
|
||||
since := 24 * time.Hour
|
||||
if q := r.URL.Query().Get("since"); q != "" {
|
||||
d, err := parseSince(q)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
since = d
|
||||
}
|
||||
from := time.Now().Add(-since)
|
||||
counts, err := h.GuardCounts(r.Context(), from)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
events, err := h.GuardEvents(r.Context(), from, 100)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
var names map[string]string
|
||||
if a.quotas != nil {
|
||||
names, _ = a.scopeNames(r.Context())
|
||||
}
|
||||
project := func(id string) string {
|
||||
if n := names["project:"+id]; n != "" {
|
||||
return n
|
||||
}
|
||||
return id
|
||||
}
|
||||
v := GuardEventsView{Since: from, Counts: []GuardCountView{}, Events: []GuardEventView{}}
|
||||
for _, c := range counts {
|
||||
v.Counts = append(v.Counts, GuardCountView{Project: project(c.ProjectID), Action: c.Action,
|
||||
Policy: c.Policy, Category: c.Category, Count: c.Count})
|
||||
}
|
||||
slices.SortStableFunc(v.Counts, func(a, b GuardCountView) int { return int(b.Count - a.Count) })
|
||||
for _, e := range events {
|
||||
v.Events = append(v.Events, GuardEventView{At: e.At, RequestID: e.RequestID, Project: project(e.ProjectID),
|
||||
Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, GuardMs: e.GuardMs})
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
})
|
||||
change := func(attach bool) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.guard == nil {
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard: in server.yaml")
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
|
||||
return
|
||||
}
|
||||
var at GuardAttachmentRequest
|
||||
@@ -118,8 +290,8 @@ func (a *API) registerGuard(mux Mux) {
|
||||
return
|
||||
}
|
||||
att.Policy = at.Policy
|
||||
if _, ok := a.guard.Library.Policies[att.Policy]; !ok && attach {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("no policy %q in server.yaml's guard.policies", att.Policy))
|
||||
if _, ok := a.guard.Library().Policies[att.Policy]; !ok && attach {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("no policy %q in the cluster's guard.policies", att.Policy))
|
||||
return
|
||||
}
|
||||
changed := true
|
||||
|
||||
@@ -36,8 +36,10 @@ func TestGuardPoliciesAreAttachedByTarget(t *testing.T) {
|
||||
st := &memGuard{}
|
||||
invalidated := 0
|
||||
a.SetGuard(&GuardAdmin{
|
||||
Library: guard.Config{Default: []string{"base"}, Policies: map[string]guard.Policy{"base": {}, "finance": {}}},
|
||||
Store: st, Invalidate: func() { invalidated++ },
|
||||
Library: func() guard.Config {
|
||||
return guard.Config{Default: []string{"base"}, Policies: map[string]guard.Policy{"base": {}, "finance": {}}}
|
||||
},
|
||||
Store: st, Invalidate: func() { invalidated++ },
|
||||
})
|
||||
if code := call(t, a, "POST", "/api/v1/guard/attachments", `{"target":"key:k1","policy":"finance"}`, nil); code != 200 {
|
||||
t.Fatalf("attach: %d", code)
|
||||
@@ -116,3 +118,55 @@ func TestUsageIsExportedAsCSVThatASpreadsheetCannotRun(t *testing.T) {
|
||||
t.Errorf("%d %s %+v", rec.Code, body, m)
|
||||
}
|
||||
}
|
||||
|
||||
// memGuardHistory is a guard store that also remembers what was done.
|
||||
type memGuardHistory struct {
|
||||
memGuard
|
||||
since time.Time
|
||||
}
|
||||
|
||||
func (m *memGuardHistory) GuardEvents(_ context.Context, since time.Time, _ int) ([]store.GuardEvent, error) {
|
||||
m.since = since
|
||||
return []store.GuardEvent{{RequestID: "r1", Model: "m", Status: 400, Action: "block", Policy: "base", Category: "S1", GuardMs: 410}}, nil
|
||||
}
|
||||
|
||||
func (m *memGuardHistory) GuardCounts(context.Context, time.Time) ([]store.GuardCount, error) {
|
||||
return []store.GuardCount{
|
||||
{Action: "flag", Policy: "base", Category: "email", Count: 2},
|
||||
{Action: "block", Policy: "base", Category: "S1", Count: 7},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func TestTheGuardrailsSayWhatEachPolicyDoesAndWhatTheyDid(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
h := &memGuardHistory{}
|
||||
lib := guard.Config{Policies: map[string]guard.Policy{
|
||||
"base": {Fail: "closed", Input: guard.Rules{PII: map[string]string{"email": "redact"}, Injection: "block"},
|
||||
Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block", Categories: []string{"S1", "S9"}}},
|
||||
}}
|
||||
a.SetGuard(&GuardAdmin{Store: h, Library: func() guard.Config { return lib }})
|
||||
|
||||
var v GuardView
|
||||
if code := call(t, a, "GET", "/api/v1/guard", "", &v); code != 200 || len(v.Details) != 1 {
|
||||
t.Fatalf("%d %+v", code, v)
|
||||
}
|
||||
d := v.Details[0]
|
||||
want := []string{"input: pii email → redact", "input: prompt injection → block", "judge llama-guard-3-1b: input → block (S1 S9)"}
|
||||
if d.Mode != "enforce" || d.Fail != "closed" || d.Judge != "llama-guard-3-1b" || !slices.Equal(d.Checks, want) {
|
||||
t.Errorf("%+v", d)
|
||||
}
|
||||
|
||||
var ev GuardEventsView
|
||||
if code := call(t, a, "GET", "/api/v1/guard/events?since=1h", "", &ev); code != 200 {
|
||||
t.Fatalf("events: %d", code)
|
||||
}
|
||||
if time.Since(h.since) > time.Hour+time.Minute || time.Since(h.since) < time.Hour-time.Minute {
|
||||
t.Errorf("asked since %s", h.since)
|
||||
}
|
||||
if len(ev.Counts) != 2 || ev.Counts[0].Count != 7 || len(ev.Events) != 1 || ev.Events[0].GuardMs != 410 {
|
||||
t.Errorf("most first, and the events: %+v", ev)
|
||||
}
|
||||
if code := call(t, a, "GET", "/api/v1/guard/events?since=soon", "", nil); code != 400 {
|
||||
t.Errorf("a bad window: %d", code)
|
||||
}
|
||||
}
|
||||
@@ -12,7 +12,7 @@ func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
|
||||
// notifyTest sends a test event to every webhook and says how each answered.
|
||||
func (a *API) notifyTest(w http.ResponseWriter, r *http.Request) {
|
||||
if a.notifier == nil {
|
||||
writeError(w, http.StatusNotFound, "no webhooks: server.yaml has no notify.webhooks")
|
||||
writeError(w, http.StatusNotFound, "no webhooks: the cluster's settings have no notify.webhooks (llmgw ctl config set notify.webhooks ...)")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.notifier.Test(r.Context()))
|
||||
|
||||
@@ -131,7 +131,7 @@ var Routes = []Route{
|
||||
Description: "With the defaults the catalog filled in. By its name only, not an alias.",
|
||||
Response: catalog.Model{}},
|
||||
{Method: "DELETE", Path: "/api/v1/models/{name}", Summary: "Remove a model added at runtime",
|
||||
Description: "A model declared in configuration is refused: it would come back on the next restart.",
|
||||
Description: "A model declared in the cluster's settings is refused: it is changed there.",
|
||||
Response: Removed{}},
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs",
|
||||
Request: ScaleRequest{}, Response: ScaleResponse{}},
|
||||
@@ -249,11 +249,29 @@ var Routes = []Route{
|
||||
Response: Removed{}},
|
||||
{Method: "GET", Path: "/api/v1/guard", Summary: "The guardrail policies and where they are attached",
|
||||
Response: GuardView{}},
|
||||
{Method: "GET", Path: "/api/v1/guard/events", Summary: "What the guardrails did: totals and the newest requests",
|
||||
Query: []Param{
|
||||
{Name: "since", Description: "How far back: 90m, 24h, 7d; 24h when absent."},
|
||||
},
|
||||
Response: GuardEventsView{}},
|
||||
{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
|
||||
Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
|
||||
{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
|
||||
Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
|
||||
|
||||
// The cluster's settings: everything in server.yaml that is not one
|
||||
// machine's own, the same on every control plane.
|
||||
{Method: "GET", Path: "/api/v1/settings", Summary: "The cluster's settings in force, or one earlier version",
|
||||
Query: []Param{{Name: "version", Description: "An earlier version; the newest when absent."}},
|
||||
Response: SettingsView{}},
|
||||
{Method: "GET", Path: "/api/v1/settings/history", Summary: "Versions of the cluster's settings, newest first",
|
||||
Query: []Param{{Name: "limit", Description: "How many; 20 when absent."}},
|
||||
Response: []SettingsView{}},
|
||||
{Method: "PUT", Path: "/api/v1/settings", Summary: "Change the cluster's settings: in force on every control plane within seconds",
|
||||
Request: SettingsRequest{}, Response: SettingsView{}},
|
||||
{Method: "POST", Path: "/api/v1/settings/rollback", Summary: "Put an earlier version of the cluster's settings back in force",
|
||||
Request: SettingsRollbackRequest{}, Response: SettingsView{}},
|
||||
|
||||
// Usage, requests and the record of changes.
|
||||
{Method: "GET", Path: "/api/v1/usage", Summary: "What each group of the tenancy tree spent",
|
||||
Query: []Param{
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// SettingsStore keeps the cluster's settings.
|
||||
type SettingsStore interface {
|
||||
ClusterSettings(ctx context.Context) (store.SettingsVersion, error)
|
||||
ClusterSettingsAt(ctx context.Context, version int64) (store.SettingsVersion, error)
|
||||
ClusterSettingsHistory(ctx context.Context, limit int) ([]store.SettingsVersion, error)
|
||||
PutClusterSettings(ctx context.Context, base int64, body, by, note string) (int64, error)
|
||||
}
|
||||
|
||||
// Settings is what the settings endpoints need: where they are kept, and a
|
||||
// check that a document would load — laid over this control plane's own
|
||||
// server.yaml, validated as it would be at start.
|
||||
type Settings struct {
|
||||
Store SettingsStore
|
||||
Check func(body []byte) error
|
||||
// LocalKeys are those each machine's server.yaml keeps, which the
|
||||
// document cannot set.
|
||||
LocalKeys []string
|
||||
}
|
||||
|
||||
// SetSettings enables the settings endpoints.
|
||||
func (a *API) SetSettings(s *Settings) { a.settings = s }
|
||||
|
||||
// SettingsView is one version of the cluster's settings.
|
||||
type SettingsView struct {
|
||||
Version int64 `json:"version"`
|
||||
Body string `json:"body"`
|
||||
Note string `json:"note,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
By string `json:"by,omitempty"`
|
||||
LocalKeys []string `json:"local_keys,omitempty"`
|
||||
}
|
||||
|
||||
// SettingsRequest is a new version of the cluster's settings, made from the
|
||||
// version Base, which must still be the newest.
|
||||
type SettingsRequest struct {
|
||||
Base int64 `json:"base"`
|
||||
Body string `json:"body"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
// SettingsRollbackRequest puts an earlier version's settings back in force,
|
||||
// as a new version.
|
||||
type SettingsRollbackRequest struct {
|
||||
Version int64 `json:"version"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
func settingsView(v store.SettingsVersion) SettingsView {
|
||||
return SettingsView{Version: v.Version, Body: v.Body, Note: v.Note, At: v.At, By: v.By}
|
||||
}
|
||||
|
||||
func (a *API) registerSettings(mux Mux) {
|
||||
need := func(w http.ResponseWriter) bool {
|
||||
if a.settings == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the cluster's settings in")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
mux.HandleFunc("GET /api/v1/settings", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
var v store.SettingsVersion
|
||||
var err error
|
||||
if q := r.URL.Query().Get("version"); q != "" {
|
||||
n, perr := strconv.ParseInt(q, 10, 64)
|
||||
if perr != nil {
|
||||
writeError(w, http.StatusBadRequest, "version is a number")
|
||||
return
|
||||
}
|
||||
v, err = a.settings.Store.ClusterSettingsAt(r.Context(), n)
|
||||
if err == nil && v.Version == 0 {
|
||||
writeError(w, http.StatusNotFound, "there is no such version")
|
||||
return
|
||||
}
|
||||
} else {
|
||||
v, err = a.settings.Store.ClusterSettings(r.Context())
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := settingsView(v)
|
||||
out.LocalKeys = a.settings.LocalKeys
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
})
|
||||
mux.HandleFunc("GET /api/v1/settings/history", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
limit := 20
|
||||
if q := r.URL.Query().Get("limit"); q != "" {
|
||||
if n, err := strconv.Atoi(q); err == nil && n > 0 && n <= 500 {
|
||||
limit = n
|
||||
}
|
||||
}
|
||||
list, err := a.settings.Store.ClusterSettingsHistory(r.Context(), limit)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]SettingsView, len(list))
|
||||
for i, v := range list {
|
||||
out[i] = settingsView(v)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
})
|
||||
put := func(w http.ResponseWriter, r *http.Request, base int64, body, note string) {
|
||||
if err := a.settings.Check([]byte(body)); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
n, err := a.settings.Store.PutClusterSettings(r.Context(), base, body, actorOf(r), note)
|
||||
switch {
|
||||
case errors.Is(err, store.ErrSettingsChanged):
|
||||
writeError(w, http.StatusConflict, err.Error())
|
||||
return
|
||||
case err != nil:
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
v, err := a.settings.Store.ClusterSettingsAt(r.Context(), n)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, settingsView(v))
|
||||
}
|
||||
mux.HandleFunc("PUT /api/v1/settings", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
var req SettingsRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
put(w, r, req.Base, req.Body, req.Note)
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/settings/rollback", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
var req SettingsRollbackRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
old, err := a.settings.Store.ClusterSettingsAt(r.Context(), req.Version)
|
||||
if err == nil && old.Version == 0 {
|
||||
writeError(w, http.StatusNotFound, "there is no such version")
|
||||
return
|
||||
}
|
||||
cur, err2 := a.settings.Store.ClusterSettings(r.Context())
|
||||
if err = errors.Join(err, err2); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
note := req.Note
|
||||
if note == "" {
|
||||
note = "back to version " + strconv.FormatInt(req.Version, 10)
|
||||
}
|
||||
put(w, r, cur.Version, old.Body, note)
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type memSettings struct{ versions []store.SettingsVersion }
|
||||
|
||||
func (m *memSettings) ClusterSettings(context.Context) (store.SettingsVersion, error) {
|
||||
if len(m.versions) == 0 {
|
||||
return store.SettingsVersion{}, nil
|
||||
}
|
||||
return m.versions[len(m.versions)-1], nil
|
||||
}
|
||||
|
||||
func (m *memSettings) ClusterSettingsAt(_ context.Context, v int64) (store.SettingsVersion, error) {
|
||||
if v < 1 || int(v) > len(m.versions) {
|
||||
return store.SettingsVersion{}, nil
|
||||
}
|
||||
return m.versions[v-1], nil
|
||||
}
|
||||
|
||||
func (m *memSettings) ClusterSettingsHistory(context.Context, int) ([]store.SettingsVersion, error) {
|
||||
var out []store.SettingsVersion
|
||||
for i := len(m.versions) - 1; i >= 0; i-- {
|
||||
out = append(out, m.versions[i])
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (m *memSettings) PutClusterSettings(_ context.Context, base int64, body, by, note string) (int64, error) {
|
||||
if base != int64(len(m.versions)) {
|
||||
return 0, store.ErrSettingsChanged
|
||||
}
|
||||
m.versions = append(m.versions, store.SettingsVersion{Version: base + 1, Body: body, By: by, Note: note, At: time.Now()})
|
||||
return base + 1, nil
|
||||
}
|
||||
|
||||
func TestSettingsAreCheckedKeptAndRolledBack(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
st := &memSettings{}
|
||||
a.SetSettings(&Settings{Store: st, Check: func(b []byte) error {
|
||||
if strings.Contains(string(b), "listen") {
|
||||
return errors.New("listen is each machine's")
|
||||
}
|
||||
return nil
|
||||
}})
|
||||
var v SettingsView
|
||||
if code := call(t, a, "PUT", "/api/v1/settings", `{"base":0,"body":"router:\n retries: 2\n"}`, &v); code != 200 || v.Version != 1 {
|
||||
t.Fatalf("first: %d %+v", code, v)
|
||||
}
|
||||
if code := call(t, a, "PUT", "/api/v1/settings", `{"base":1,"body":"listen: :1\n"}`, nil); code != 400 {
|
||||
t.Errorf("a document the check refuses: %d", code)
|
||||
}
|
||||
if code := call(t, a, "PUT", "/api/v1/settings", `{"base":0,"body":"router: {}\n"}`, nil); code != 409 {
|
||||
t.Errorf("a change from an old version: %d", code)
|
||||
}
|
||||
if code := call(t, a, "PUT", "/api/v1/settings", `{"base":1,"body":"router:\n retries: 5\n"}`, nil); code != 200 {
|
||||
t.Fatalf("second: %d", code)
|
||||
}
|
||||
if code := call(t, a, "POST", "/api/v1/settings/rollback", `{"version":1}`, &v); code != 200 ||
|
||||
v.Version != 3 || v.Body != "router:\n retries: 2\n" || v.Note != "back to version 1" {
|
||||
t.Errorf("rollback: %d %+v", code, v)
|
||||
}
|
||||
var h []SettingsView
|
||||
if code := call(t, a, "GET", "/api/v1/settings/history", "", &h); code != 200 || len(h) != 3 || h[0].Version != 3 {
|
||||
t.Errorf("history: %d %+v", code, h)
|
||||
}
|
||||
if code := call(t, a, "GET", "/api/v1/settings?version=9", "", nil); code != 404 {
|
||||
t.Errorf("a version that is not there: %d", code)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
@@ -14,8 +15,11 @@ import (
|
||||
// derived from the same state the tables show, so it can never disagree with
|
||||
// them; it is a different question asked of the same facts.
|
||||
type Summary struct {
|
||||
Nodes NodeTotals `json:"nodes"`
|
||||
GPUs []ClassUse `json:"gpus"`
|
||||
// Version is the release the answering control plane runs: not the
|
||||
// binary on its disk, which a manual install changes underneath it.
|
||||
Version string `json:"version,omitempty"`
|
||||
Nodes NodeTotals `json:"nodes"`
|
||||
GPUs []ClassUse `json:"gpus"`
|
||||
// Fabrics matter because a multi-node replica may never span two of them,
|
||||
// so free capacity in the wrong fabric is not capacity for the model that
|
||||
// needs it.
|
||||
@@ -35,6 +39,9 @@ type Summary struct {
|
||||
// StoreError why it could not be read; both empty for a single database.
|
||||
Store []StoreMember `json:"store,omitempty"`
|
||||
StoreError string `json:"store_error,omitempty"`
|
||||
// ControlPlanes is the cluster's control planes when there is more than
|
||||
// one: which leads, which stand by, and the floating address in front.
|
||||
ControlPlanes ControlPlanes `json:"control_planes,omitzero"`
|
||||
}
|
||||
|
||||
// StoreMember is one member of a replicated store, as its Patroni reports it.
|
||||
@@ -47,6 +54,27 @@ type StoreMember struct {
|
||||
Lag int64 `json:"lag"`
|
||||
}
|
||||
|
||||
// ControlPlanes is every control plane of a cluster that has several, as
|
||||
// each answered when asked.
|
||||
type ControlPlanes struct {
|
||||
Members []ControlPlane `json:"members,omitempty"`
|
||||
// Floating is the address keepalived keeps on the leader (10.0.0.161),
|
||||
// and FloatingUp whether it answered as a leader when asked; empty when
|
||||
// nothing floats.
|
||||
Floating string `json:"floating,omitempty"`
|
||||
FloatingUp bool `json:"floating_up,omitempty"`
|
||||
}
|
||||
|
||||
// ControlPlane is one control plane.
|
||||
type ControlPlane struct {
|
||||
Name string `json:"name"`
|
||||
Address string `json:"address"`
|
||||
// State is "leading", "standing by" or "not answering".
|
||||
State string `json:"state"`
|
||||
// This is the control plane that answered the summary.
|
||||
This bool `json:"this,omitempty"`
|
||||
}
|
||||
|
||||
// NodeTotals counts nodes by the states that need different responses.
|
||||
type NodeTotals struct {
|
||||
Total int `json:"total"`
|
||||
@@ -120,7 +148,7 @@ const idleThreshold = 10 * time.Minute
|
||||
|
||||
// Summary builds the fleet summary.
|
||||
func (a *API) Summary() Summary {
|
||||
var s Summary
|
||||
s := Summary{Version: version.Version}
|
||||
if a.storeStatus != nil {
|
||||
members, err := a.storeStatus()
|
||||
s.Store = members
|
||||
@@ -128,6 +156,9 @@ func (a *API) Summary() Summary {
|
||||
s.StoreError = err.Error()
|
||||
}
|
||||
}
|
||||
if a.controlPlanes != nil {
|
||||
s.ControlPlanes = a.controlPlanes()
|
||||
}
|
||||
|
||||
nodes := a.Nodes()
|
||||
byClass := map[string]*ClassUse{}
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// SetFirewall gives the service whether nodes keep llmgw's firewall, read
|
||||
// at each use so a change to the cluster's settings is what is sent next.
|
||||
func (s *Service) SetFirewall(f func() *llmgwv1.Firewall) {
|
||||
s.mu.Lock()
|
||||
s.firewall = f
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Service) firewallMsg() *llmgwv1.ControlMessage {
|
||||
s.mu.RLock()
|
||||
f := s.firewall
|
||||
s.mu.RUnlock()
|
||||
if f == nil {
|
||||
return nil
|
||||
}
|
||||
fw := f()
|
||||
if fw == nil {
|
||||
return nil
|
||||
}
|
||||
return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Firewall{Firewall: fw}}
|
||||
}
|
||||
|
||||
// BroadcastFirewall tells every connected agent what the cluster's settings
|
||||
// now say of the firewall. A node not connected is told when it registers.
|
||||
func (s *Service) BroadcastFirewall() {
|
||||
msg := s.firewallMsg()
|
||||
if msg == nil {
|
||||
return
|
||||
}
|
||||
for _, node := range s.ConnectedNodes() {
|
||||
if err := s.send(node, msg); err != nil {
|
||||
s.log.Warn("could not tell a node about the firewall; it is told when it next registers",
|
||||
"node", node, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -75,6 +75,8 @@ type Service struct {
|
||||
|
||||
// controlPlanes lists what agents may dial; see SetControlPlanes.
|
||||
controlPlanes func(context.Context) ([]string, error)
|
||||
// firewall is what nodes are told of llmgw's firewall; see SetFirewall.
|
||||
firewall func() *llmgwv1.Firewall
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -286,6 +288,12 @@ func (s *Service) Connect(
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Whether to keep llmgw's firewall, as the cluster's settings say now.
|
||||
if msg := s.firewallMsg(); msg != nil {
|
||||
if err := stream.Send(msg); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Recover replica state from what this node is already running, before any
|
||||
// reconciliation pass can conclude the cluster is empty and place
|
||||
@@ -302,16 +310,58 @@ func (s *Service) Connect(
|
||||
}()
|
||||
defer wg.Wait()
|
||||
|
||||
// Read in a goroutine of its own, so the stream can also be ended from
|
||||
// this side: Receive blocks until the agent sends, and an agent that is
|
||||
// only heartbeating would keep a stream this process should no longer
|
||||
// hold for as long as it likes. Returning ends it; the reader's Receive
|
||||
// then fails and it follows.
|
||||
read := make(chan struct{})
|
||||
go func() {
|
||||
defer close(read)
|
||||
s.readLoop(ctx, stream, c, nodeID)
|
||||
}()
|
||||
tick := time.NewTicker(leaderCheck)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-read:
|
||||
return nil
|
||||
case <-c.closed:
|
||||
// Replaced by a newer stream from the same node, or the writer
|
||||
// failed.
|
||||
return nil
|
||||
case <-tick.C:
|
||||
// Leadership lost while the process lives on — the store was
|
||||
// slow, or another took the lease. A standby holding agents is
|
||||
// a cluster whose leader cannot reach its nodes, so let them go
|
||||
// and they find the leader.
|
||||
if s.leader != nil && !s.leader.IsLeader() {
|
||||
s.log.Info("no longer leading; letting the agent go", "node", nodeID)
|
||||
c.close()
|
||||
return connect.NewError(connect.CodeUnavailable,
|
||||
errors.New("this control plane is a standby; connect to the leader"))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// leaderCheck is how often an open stream checks that this process still
|
||||
// leads.
|
||||
var leaderCheck = time.Second
|
||||
|
||||
func (s *Service) readLoop(ctx context.Context,
|
||||
stream *connect.BidiStream[llmgwv1.AgentMessage, llmgwv1.ControlMessage], c *conn, nodeID string,
|
||||
) {
|
||||
for {
|
||||
msg, err := stream.Receive()
|
||||
if err != nil {
|
||||
c.close()
|
||||
if ctx.Err() != nil || errors.Is(err, io.EOF) {
|
||||
s.log.Info("agent disconnected", "node", nodeID)
|
||||
return nil
|
||||
return
|
||||
}
|
||||
s.log.Warn("agent stream failed", "node", nodeID, "err", err)
|
||||
return nil
|
||||
return
|
||||
}
|
||||
if kv := msg.GetKvEvents(); kv != nil {
|
||||
s.kvEvents(c, kv)
|
||||
|
||||
@@ -231,6 +231,10 @@ func (c *Controller) SetLeadership(l *Leadership) {
|
||||
c.rec.SetLeadership(l)
|
||||
}
|
||||
|
||||
// Leadership is what SetLeadership set; nil when there is nobody to compete
|
||||
// with.
|
||||
func (c *Controller) Leadership() *Leadership { return c.lead }
|
||||
|
||||
// IsLeader reports whether this process is placing work. True when leadership
|
||||
// was never configured, because then there is nobody to compete with.
|
||||
func (c *Controller) IsLeader() bool { return c.lead == nil || c.lead.IsLeader() }
|
||||
|
||||
@@ -43,6 +43,39 @@ func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logge
|
||||
return c
|
||||
}
|
||||
|
||||
// Reload reads the choices again, as a standby becoming the leader does, and
|
||||
// tells each node whose choice changed.
|
||||
func (c *EngineChoices) Reload(ctx context.Context) {
|
||||
got, err := c.st.NodeEngines(ctx)
|
||||
if err != nil {
|
||||
c.log.Error("could not read the engine builds chosen per node again; keeping the ones read before", "err", err)
|
||||
return
|
||||
}
|
||||
c.mu.Lock()
|
||||
var changed []string
|
||||
for node := range got {
|
||||
if !maps.Equal(got[node], c.byNode[node]) {
|
||||
changed = append(changed, node)
|
||||
}
|
||||
}
|
||||
for node := range c.byNode {
|
||||
if _, ok := got[node]; !ok {
|
||||
changed = append(changed, node)
|
||||
}
|
||||
}
|
||||
c.byNode = got
|
||||
notify := c.notify
|
||||
c.mu.Unlock()
|
||||
if notify == nil {
|
||||
return
|
||||
}
|
||||
for _, node := range changed {
|
||||
if err := notify(node); err != nil {
|
||||
c.log.Warn("could not tell a node its engine builds", "node", node, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// OnChange is called with a node whose choice changed, to tell it.
|
||||
func (c *EngineChoices) OnChange(f func(node string) error) {
|
||||
c.mu.Lock()
|
||||
|
||||
@@ -68,6 +68,23 @@ func (f *Facility) UseStore(ctx context.Context, st FacilityStore) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.st = st
|
||||
f.loadLocked(ctx)
|
||||
}
|
||||
|
||||
// Reload reads the holds and labels in the store again, as a standby
|
||||
// becoming the leader does.
|
||||
func (f *Facility) Reload(ctx context.Context) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.st == nil {
|
||||
return
|
||||
}
|
||||
f.holds = map[facility.Domain]facility.Hold{}
|
||||
f.loadLocked(ctx)
|
||||
}
|
||||
|
||||
func (f *Facility) loadLocked(ctx context.Context) {
|
||||
st := f.st
|
||||
if holds, err := st.FacilityHolds(ctx); err != nil {
|
||||
f.log.Error("could not read the domains held for maintenance; none is held until they are set again", "err", err)
|
||||
} else {
|
||||
|
||||
@@ -7,6 +7,9 @@ import (
|
||||
"errors"
|
||||
"log/slog"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
@@ -51,6 +54,29 @@ type Leadership struct {
|
||||
renew time.Duration
|
||||
|
||||
leader atomic.Bool
|
||||
// stopped is set by StepDown: nothing takes the lease again after it.
|
||||
stopped atomic.Bool
|
||||
|
||||
hookMu sync.Mutex
|
||||
onLead []func()
|
||||
}
|
||||
|
||||
// OnLead is called each time this process becomes the leader, after
|
||||
// standing by: what it read at start may have been changed since by the
|
||||
// leader before it, and is read again.
|
||||
func (l *Leadership) OnLead(f func()) {
|
||||
l.hookMu.Lock()
|
||||
l.onLead = append(l.onLead, f)
|
||||
l.hookMu.Unlock()
|
||||
}
|
||||
|
||||
func (l *Leadership) becameLeader() {
|
||||
l.hookMu.Lock()
|
||||
hooks := slices.Clone(l.onLead)
|
||||
l.hookMu.Unlock()
|
||||
for _, f := range hooks {
|
||||
go f()
|
||||
}
|
||||
}
|
||||
|
||||
// NewLeadership prepares leadership over a store.
|
||||
@@ -117,11 +143,20 @@ func (l *Leadership) Run(ctx context.Context) {
|
||||
// process: a line per attempt would bury everything else, and the thing an
|
||||
// operator needs to see is the moment leadership moved.
|
||||
func (l *Leadership) attempt(ctx context.Context) {
|
||||
if l.stopped.Load() {
|
||||
return
|
||||
}
|
||||
lease, err := l.store.AcquireLeadership(ctx, controlRole, l.owner, l.ttl)
|
||||
switch {
|
||||
case err == nil && l.stopped.Load():
|
||||
// Stepped down while this was on its way to the store: give back
|
||||
// what it just took.
|
||||
l.leader.Store(true)
|
||||
l.step(ctx)
|
||||
case err == nil:
|
||||
if l.leader.CompareAndSwap(false, true) {
|
||||
l.log.Info("this control plane is now placing work", "owner", l.owner)
|
||||
l.becameLeader()
|
||||
}
|
||||
case errors.Is(err, store.ErrNotLeader):
|
||||
if l.leader.CompareAndSwap(true, false) {
|
||||
@@ -145,10 +180,24 @@ func (l *Leadership) attempt(ctx context.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
// StepDown gives the lease up now, for a process on its way out: called
|
||||
// before the agents are let go, so the standby they move to is already the
|
||||
// leader, and before the process exits, which would otherwise race the
|
||||
// release in Run and leave the standby to wait out the lease. Run's own
|
||||
// release after it is then nothing.
|
||||
func (l *Leadership) StepDown() {
|
||||
if l.store == nil {
|
||||
return
|
||||
}
|
||||
l.stopped.Store(true)
|
||||
l.step(context.Background())
|
||||
}
|
||||
|
||||
// step gives the lease up on the way out, so a standby takes over in seconds
|
||||
// rather than waiting out the clock.
|
||||
func (l *Leadership) step(ctx context.Context) {
|
||||
if !l.leader.Load() {
|
||||
// Once: StepDown and Run's own way out both come here.
|
||||
if !l.leader.CompareAndSwap(true, false) {
|
||||
return
|
||||
}
|
||||
// A fresh context: the one that just ended is the reason this is running.
|
||||
@@ -180,3 +229,16 @@ func newOwnerID() string {
|
||||
}
|
||||
return host + "-" + hex.EncodeToString(b[:])
|
||||
}
|
||||
|
||||
// ControlRole is the lease's name in the store, for readers that ask who
|
||||
// holds it without competing.
|
||||
const ControlRole = controlRole
|
||||
|
||||
// HolderName is the machine an owner id names: the hostname it was made
|
||||
// from, without the random part.
|
||||
func HolderName(owner string) string {
|
||||
if i := strings.LastIndexByte(owner, '-'); i > 0 {
|
||||
return owner[:i]
|
||||
}
|
||||
return owner
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestAStoppingLeaderHandsOverAtOnceAndNeverTakesItBack(t *testing.T) {
|
||||
st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
log := slog.New(slog.DiscardHandler)
|
||||
a := NewLeadership(st, time.Minute, log)
|
||||
b := NewLeadership(st, time.Minute, log)
|
||||
|
||||
a.attempt(t.Context())
|
||||
b.attempt(t.Context())
|
||||
if !a.IsLeader() || b.IsLeader() {
|
||||
t.Fatalf("a %v, b %v", a.IsLeader(), b.IsLeader())
|
||||
}
|
||||
|
||||
// Stepping down gives the lease back, rather than leaving b to wait out
|
||||
// the minute it was taken for.
|
||||
a.StepDown()
|
||||
if a.IsLeader() {
|
||||
t.Error("still leading after stepping down")
|
||||
}
|
||||
// A renewal already on its way when the process began to stop must not
|
||||
// take it back.
|
||||
a.attempt(t.Context())
|
||||
if a.IsLeader() {
|
||||
t.Error("took the lease again after stepping down")
|
||||
}
|
||||
b.attempt(t.Context())
|
||||
if !b.IsLeader() {
|
||||
t.Error("the standby could not take the lease that was given up")
|
||||
}
|
||||
lease, err := st.Leadership(t.Context(), ControlRole)
|
||||
if err != nil || lease.Owner != b.Owner() {
|
||||
t.Errorf("the lease is %q (%v), not b's", lease.Owner, err)
|
||||
}
|
||||
if got := HolderName(b.Owner()); got == "" || got == b.Owner() {
|
||||
t.Errorf("holder name %q from %q", got, b.Owner())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStandbyIsToldWhenItBecomesTheLeader(t *testing.T) {
|
||||
st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
log := slog.New(slog.DiscardHandler)
|
||||
a := NewLeadership(st, time.Minute, log)
|
||||
b := NewLeadership(st, time.Minute, log)
|
||||
led := make(chan struct{}, 4)
|
||||
b.OnLead(func() { led <- struct{}{} })
|
||||
ctx := t.Context()
|
||||
a.attempt(ctx)
|
||||
b.attempt(ctx)
|
||||
if !a.IsLeader() || b.IsLeader() {
|
||||
t.Fatal("a should lead, b stand by")
|
||||
}
|
||||
a.StepDown()
|
||||
b.attempt(ctx)
|
||||
select {
|
||||
case <-led:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("b became the leader and was not told")
|
||||
}
|
||||
}
|
||||
@@ -140,9 +140,14 @@ type Request struct {
|
||||
// llamaguard3, shieldgemma or json. Empty serves clients.
|
||||
GuardFormat string `json:"guard_format,omitempty"`
|
||||
// CtxPool is llama.cpp's --ctx-size; ContextLength what one request may
|
||||
// use. They default to 16384 and 4096, what four parallel slots divide.
|
||||
// use. They default to 16384 and 4096, what four parallel slots divide;
|
||||
// llama.cpp is given as many slots as the pool holds requests.
|
||||
CtxPool int `json:"ctx_pool,omitempty"`
|
||||
ContextLength int `json:"context_length,omitempty"`
|
||||
// Tools says the model makes tool calls the engine returns as calls: it
|
||||
// is written into the catalog, where clients and editors read it. Check
|
||||
// a running replica with llmgw ctl tool-check before saying so.
|
||||
Tools bool `json:"tools,omitempty"`
|
||||
// CPUCores is what a replica computes on; zero takes half the smallest
|
||||
// node it fits on.
|
||||
CPUCores int `json:"cpu_cores,omitempty"`
|
||||
@@ -261,6 +266,10 @@ func (s *Server) Start(req Request) (Job, error) {
|
||||
if req.ContextLength <= 0 {
|
||||
req.ContextLength = 4096
|
||||
}
|
||||
if req.ContextLength > req.CtxPool {
|
||||
return Job{}, fmt.Errorf("a request of %d tokens cannot fit a pool of %d: give --ctx-pool at least --context-length",
|
||||
req.ContextLength, req.CtxPool)
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.seq++
|
||||
j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
|
||||
@@ -545,12 +554,12 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
||||
switch {
|
||||
case o.CPU:
|
||||
p.Name = "cpu"
|
||||
p.Args = []string{"--ctx-size", strconv.Itoa(req.CtxPool)}
|
||||
p.Args = llamaArgs(req)
|
||||
p.MinMemoryGB, p.CPUCores = memGB, cores
|
||||
case o.Engine == "llamacpp":
|
||||
p.Name = fmt.Sprintf("gpu-%dx%s", o.GPUsPerNode, o.Class)
|
||||
p.GPUClass, p.GPUsPerNode = o.Class, o.GPUsPerNode
|
||||
p.Args = []string{"--ctx-size", strconv.Itoa(req.CtxPool)}
|
||||
p.Args = llamaArgs(req)
|
||||
default:
|
||||
p.Name = fmt.Sprintf("gpu-%dx%s", o.GPUs(), o.Class)
|
||||
p.GPUClass, p.TP = o.Class, o.GPUsPerNode
|
||||
@@ -564,6 +573,23 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
||||
return out
|
||||
}
|
||||
|
||||
// llamaArgs size llama.cpp's pool and its slots together. The pool is shared
|
||||
// by the slots, so slots of the context the catalog promises are as many as
|
||||
// the pool holds: the engine's default of four turned a 32768 pool into
|
||||
// slots of 8192, and the gateway, told 32768, sent requests the engine then
|
||||
// refused.
|
||||
func llamaArgs(req Request) []string {
|
||||
return []string{"--ctx-size", strconv.Itoa(req.CtxPool),
|
||||
"--parallel", strconv.Itoa(max(1, req.CtxPool/req.ContextLength))}
|
||||
}
|
||||
|
||||
// maxOutput is the longest answer the catalog allows: a quarter of the
|
||||
// context, from 1024 to 8192. 1024 everywhere cut an editor's agent off in
|
||||
// the middle of the file it was writing.
|
||||
func maxOutput(contextLength int) int {
|
||||
return min(8192, max(1024, contextLength/4))
|
||||
}
|
||||
|
||||
// addAndStart puts the model in the catalog with its profile, waits for a
|
||||
// replica and asks it to answer.
|
||||
// hardwareExists is whether the cluster has what the ways chosen need, busy
|
||||
@@ -583,10 +609,10 @@ func (s *Server) addAndStart(ctx context.Context, j *Job, repo hf.Repo, set hf.F
|
||||
// 4. The catalog.
|
||||
m := catalog.Model{
|
||||
Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
|
||||
RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
|
||||
RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: maxOutput(req.ContextLength),
|
||||
Profiles: profs,
|
||||
Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
|
||||
Capabilities: catalog.Capabilities{Vision: req.Projector != ""},
|
||||
Capabilities: catalog.Capabilities{Vision: req.Projector != "", Tools: req.Tools},
|
||||
Origin: &catalog.Origin{Repo: repo.ID, Revision: repo.Revision, Files: originFiles(set)},
|
||||
}
|
||||
if req.GuardFormat != "" {
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
package modelserve
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestLlamaCppIsGivenTheSlotsTheCatalogPromises(t *testing.T) {
|
||||
// A 32768 pool for 32768-token requests is one slot: the engine's own
|
||||
// four made slots of 8192, and requests the gateway admitted were refused.
|
||||
for _, c := range []struct {
|
||||
pool, ctx int
|
||||
want []string
|
||||
}{
|
||||
{32768, 32768, []string{"--ctx-size", "32768", "--parallel", "1"}},
|
||||
{16384, 4096, []string{"--ctx-size", "16384", "--parallel", "4"}},
|
||||
{65536, 16384, []string{"--ctx-size", "65536", "--parallel", "4"}},
|
||||
} {
|
||||
if got := llamaArgs(Request{CtxPool: c.pool, ContextLength: c.ctx}); !slices.Equal(got, c.want) {
|
||||
t.Errorf("pool %d, context %d: %v", c.pool, c.ctx, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheLongestAnswerGrowsWithTheContext(t *testing.T) {
|
||||
for ctx, want := range map[int]int{2048: 1024, 4096: 1024, 16384: 4096, 32768: 8192, 131072: 8192} {
|
||||
if got := maxOutput(ctx); got != want {
|
||||
t.Errorf("context %d: max output %d, want %d", ctx, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -109,7 +109,7 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
|
||||
defer s.mu.Unlock()
|
||||
|
||||
if s.Declared(m.Name) {
|
||||
return fmt.Errorf("model %q is declared in configuration; edit that rather than adding a second definition", m.Name)
|
||||
return fmt.Errorf("model %q is declared in the cluster's settings; edit it there (llmgw ctl config edit) rather than adding a second definition", m.Name)
|
||||
}
|
||||
// A model with no GPU is all host memory, and one that declares none
|
||||
// fits everywhere as far as placement can tell — however many of it
|
||||
@@ -170,7 +170,7 @@ func (s *Service) Remove(ctx context.Context, name string) error {
|
||||
defer s.mu.Unlock()
|
||||
|
||||
if s.Declared(name) {
|
||||
return fmt.Errorf("model %q is declared in configuration; remove it there, or it comes back on the next restart", name)
|
||||
return fmt.Errorf("model %q is declared in the cluster's settings, under models; remove it there: llmgw ctl config edit", name)
|
||||
}
|
||||
|
||||
stored, err := s.storedModels(ctx)
|
||||
@@ -368,7 +368,7 @@ func (s *Service) update(ctx context.Context, name, by, what string, change func
|
||||
|
||||
if s.Declared(name) {
|
||||
return catalog.Model{}, fmt.Errorf(
|
||||
"model %q is declared in configuration; %s there, or it reverts on the next restart", name, what)
|
||||
"model %q is declared in the cluster's settings, under models; %s there: llmgw ctl config edit", name, what)
|
||||
}
|
||||
stored, err := s.storedModels(ctx)
|
||||
if err != nil {
|
||||
|
||||
@@ -96,7 +96,7 @@ func TestConfigurationWinsOverTheDatabase(t *testing.T) {
|
||||
if err == nil {
|
||||
t.Fatal("a stored model was allowed to shadow a declared one")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "declared in configuration") {
|
||||
if !strings.Contains(err.Error(), "declared in the cluster's settings") {
|
||||
t.Errorf("the error does not say why: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -109,7 +109,7 @@ func TestADeclaredModelCannotBeRemovedAtRuntime(t *testing.T) {
|
||||
if err == nil {
|
||||
t.Fatal("a declared model was removed at runtime")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "comes back") {
|
||||
if !strings.Contains(err.Error(), "llmgw ctl config edit") {
|
||||
t.Errorf("the error does not explain the consequence: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -264,7 +264,7 @@ func TestADeclaredModelCannotBeScaledHere(t *testing.T) {
|
||||
if err == nil {
|
||||
t.Fatal("expected scaling a declared model to be refused")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "configuration") {
|
||||
if !strings.Contains(err.Error(), "cluster's settings") {
|
||||
t.Errorf("the error should say where to change it; got %q", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -117,6 +117,9 @@ var Reference = []Block{
|
||||
{Key: "name", Type: "string", Doc: "Name is this cluster as a peer reads it, in the federated_from tag\nsent with a request to a peer set to receive tags (`ctl federation add\n--send-tags`). Empty is this machine's host name.", Nested: ""},
|
||||
{Key: "interval", Type: "time.Duration", Doc: "Interval is how often each peer is asked whether it is ready, timed,\nand what it offers. A peer that does not answer is taken out of\nrouting until it does. Zero is 10s.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "FirewallSettings", Title: "FirewallSettings", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled keeps llmgw's internal ports — engines, node exporter, the\nagents' stream, the mirror, the store — to the cluster's networks on\nevery machine. False removes llmgw's nftables table everywhere, for a\nsite whose own firewall does that job; the rules are kept, and true\nputs them back. Changed in the cluster's settings, it is in force on\nevery node and control plane within seconds.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "GuardModel", Title: "GuardModel", Doc: "", Fields: []Field{
|
||||
{Key: "format", Type: "string", Doc: "Format is llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and\nthe categories), shieldgemma (Google's ShieldGemma: Yes or No), or\njson ({\"safe\": false, \"categories\": [\"...\"]}).", Nested: ""},
|
||||
{Key: "categories", Type: "[]string", Doc: "Categories the model is asked about, for shieldgemma — each a policy\nsentence — and the names reported for llamaguard3's S1, S2, ….", Nested: ""},
|
||||
@@ -363,6 +366,7 @@ var Reference = []Block{
|
||||
{Key: "guard", Type: "guard.Config", Doc: "Guard is the library of guardrail policies: what requests and answers\nare checked for, and what is done. Applied to every request through\nguard.default, and to parts of the tenancy tree by attaching them\n(llmgw ctl guard attach). Empty checks nothing.", Nested: "guard.Config"},
|
||||
{Key: "secrets", Type: "Secrets", Doc: "Secrets is where secrets kept outside files are read from, and the key\nthe store's own secrets are sealed under.", Nested: "config.Secrets"},
|
||||
{Key: "notify", Type: "notify.Config", Doc: "Notify sends what needs a person — a model parked, a node lost, a\nbudget spent, new weights published — to webhooks, signed, without a\nPrometheus and Alertmanager in between. Empty sends nothing.", Nested: "notify.Config"},
|
||||
{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
|
||||
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
|
||||
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
|
||||
{Key: "backup_copy_to", Type: "[]string", Doc: "BackupCopyTo is where each archive is copied once written, so it\nsurvives this machine: a directory (a mounted share, another disk) or\nssh://user@host[:port]/path, reached with BackupCopyKey and checked\nagainst /etc/llmgw/backup_known_hosts, which learns a host the first\ntime. The same number of archives is kept there as here. Refused\nwithout backup_recipients: an archive leaves the machine encrypted.", Nested: ""},
|
||||
|
||||
@@ -30,6 +30,7 @@ var Sources = []string{
|
||||
"internal/guard/policy.go",
|
||||
"internal/guard/judge.go",
|
||||
"internal/config/oidc.go",
|
||||
"internal/config/firewall.go",
|
||||
"internal/notify/notify.go",
|
||||
"internal/secrets/secrets.go",
|
||||
"internal/obs/trace.go",
|
||||
|
||||
@@ -11,10 +11,13 @@
|
||||
package firewall
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"os"
|
||||
"os/exec"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -317,3 +320,87 @@ func Present(ctx context.Context, r Runner) bool {
|
||||
_, err := r.Run(ctx, "sudo -n /usr/sbin/nft list table inet "+Table+" >/dev/null 2>&1", nil)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// Admit lets addresses through a machine's ruleset when it has one: each
|
||||
// one's network, as Networks finds it, unless a network already there holds
|
||||
// it. A machine without llmgw's rules is left without them. What was added
|
||||
// is returned, nothing when every address was already let in.
|
||||
func Admit(ctx context.Context, r Runner, addrs []string) ([]netip.Prefix, error) {
|
||||
old, err := r.Run(ctx, "sudo cat "+RulesFile+" 2>/dev/null || true", nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
prev := parse(string(old))
|
||||
if len(prev.Ports) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
var missing []string
|
||||
for _, s := range addrs {
|
||||
a, err := netip.ParseAddr(s)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if !slices.ContainsFunc(prev.Allow, func(p netip.Prefix) bool { return p.Contains(a) }) {
|
||||
missing = append(missing, s)
|
||||
}
|
||||
}
|
||||
if len(missing) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
nets, err := Networks(ctx, r, missing)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := Install(ctx, r, Spec{Allow: nets}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return nets, nil
|
||||
}
|
||||
|
||||
// Disable takes llmgw's table out of a machine's ruleset and stops it being
|
||||
// loaded at boot, keeping the rules file: Install, or enabling the unit,
|
||||
// puts it back as it was. Reports whether there was anything to take out.
|
||||
func Disable(ctx context.Context, r Runner) (bool, error) {
|
||||
had := Present(ctx, r)
|
||||
out, err := r.Run(ctx, `sudo systemctl disable --now `+Unit+` >/dev/null 2>&1 || true
|
||||
if sudo /usr/sbin/nft list table inet `+Table+` >/dev/null 2>&1; then
|
||||
sudo /usr/sbin/nft delete table inet `+Table+`
|
||||
fi`, nil)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return had, nil
|
||||
}
|
||||
|
||||
// Enable loads the rules file a machine already has and has it loaded at
|
||||
// boot again, after Disable. Reports false when there is no rules file.
|
||||
func Enable(ctx context.Context, r Runner) (bool, error) {
|
||||
out, err := r.Run(ctx, `[ -f `+RulesFile+` ] || { echo none; exit 0; }
|
||||
sudo systemctl enable `+Unit+` >/dev/null 2>&1
|
||||
sudo /usr/sbin/nft -f `+RulesFile+`
|
||||
sudo systemctl start `+Unit+` >/dev/null 2>&1 || true
|
||||
echo loaded`, nil)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return strings.TrimSpace(string(out)) == "loaded", nil
|
||||
}
|
||||
|
||||
// Local runs scripts on this machine. As root, sudo in them runs its command
|
||||
// directly, for a machine that has no sudo.
|
||||
type Local struct{}
|
||||
|
||||
// Run runs script with sh.
|
||||
func (Local) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
|
||||
if os.Geteuid() == 0 {
|
||||
script = "sudo() { \"$@\"; }\n" + script
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, "sh", "-c", script)
|
||||
if stdin != nil {
|
||||
cmd.Stdin = bytes.NewReader(stdin)
|
||||
}
|
||||
var out bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &out
|
||||
err := cmd.Run()
|
||||
return out.Bytes(), err
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package firewall
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -58,3 +59,54 @@ func TestARulesetReadsBackAsItsSpec(t *testing.T) {
|
||||
t.Errorf("read back as\n%s\nfrom\n%s", Script(got), Script(in))
|
||||
}
|
||||
}
|
||||
|
||||
// fakeMachine answers the scripts Admit runs: the ruleset file, the route,
|
||||
// and an install, which it records.
|
||||
type fakeMachine struct {
|
||||
rules string
|
||||
route string
|
||||
installed []byte
|
||||
}
|
||||
|
||||
func (f *fakeMachine) Run(_ context.Context, script string, stdin []byte) ([]byte, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(script, "sudo cat "):
|
||||
return []byte(f.rules), nil
|
||||
case strings.HasPrefix(script, "ip -o route get"):
|
||||
return []byte(f.route), nil
|
||||
default:
|
||||
f.installed = stdin
|
||||
return []byte("changed\n"), nil
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeOnAnotherNetworkIsLetIn(t *testing.T) {
|
||||
rules := Script(Spec{Ports: []Port{AgentPort, MirrorFiles},
|
||||
Allow: []netip.Prefix{netip.MustParsePrefix("10.104.130.128/25")}})
|
||||
m := &fakeMachine{rules: rules,
|
||||
route: "10.20.0.5 via 10.104.130.129 dev eth0 src 10.104.130.140 uid 0\n" +
|
||||
"2: eth0 inet 10.104.130.140/25 brd 10.104.130.255 scope global eth0\n"}
|
||||
|
||||
added, err := Admit(context.Background(), m, []string{"10.104.130.200"})
|
||||
if err != nil || len(added) != 0 || m.installed != nil {
|
||||
t.Fatalf("an address the rules hold: added %v, err %v, installed %v", added, err, m.installed != nil)
|
||||
}
|
||||
|
||||
added, err = Admit(context.Background(), m, []string{"10.20.0.5", "gpu01"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(added) != 1 || added[0].String() != "10.20.0.5/32" {
|
||||
t.Errorf("added %v, want the routed node's address alone", added)
|
||||
}
|
||||
got := parse(strings.Split(string(m.installed), "\x00")[0])
|
||||
if len(got.Ports) != 2 || len(got.Allow) != 2 {
|
||||
t.Errorf("installed %+v, want both ports kept and both networks", got)
|
||||
}
|
||||
|
||||
// No llmgw rules on the machine: none are made.
|
||||
bare := &fakeMachine{}
|
||||
if added, err := Admit(context.Background(), bare, []string{"10.20.0.5"}); err != nil || added != nil || bare.installed != nil {
|
||||
t.Errorf("a machine without rules: added %v, err %v", added, err)
|
||||
}
|
||||
}
|
||||
+44
-6
@@ -9,6 +9,7 @@ import (
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard/detect"
|
||||
@@ -36,7 +37,7 @@ type Gate struct {
|
||||
log *slog.Logger
|
||||
now func() time.Time
|
||||
|
||||
deny map[string]*regexp.Regexp // by pattern regex
|
||||
deny atomic.Pointer[map[string]*regexp.Regexp] // by pattern regex
|
||||
tools toolMemo
|
||||
judger *judge
|
||||
m *gateMetrics
|
||||
@@ -59,16 +60,53 @@ func NewGate(cfg Config, store Attachments, log *slog.Logger) (*Gate, error) {
|
||||
log = slog.Default()
|
||||
}
|
||||
g := &Gate{cfg: cfg, store: store, log: log, now: time.Now,
|
||||
deny: map[string]*regexp.Regexp{}, merged: map[string]Policy{}, m: newGateMetrics()}
|
||||
merged: map[string]Policy{}, m: newGateMetrics()}
|
||||
g.deny.Store(compileDeny(cfg))
|
||||
g.reload(context.Background())
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func compileDeny(cfg Config) *map[string]*regexp.Regexp {
|
||||
deny := map[string]*regexp.Regexp{}
|
||||
for _, p := range cfg.Policies {
|
||||
for _, r := range []Rules{p.Input, p.Output} {
|
||||
for _, d := range r.Deny {
|
||||
g.deny[d.Regex] = regexp.MustCompile(d.Regex)
|
||||
deny[d.Regex] = regexp.MustCompile(d.Regex)
|
||||
}
|
||||
}
|
||||
}
|
||||
return &deny
|
||||
}
|
||||
|
||||
// Replace puts a new library of policies in force, for requests from the
|
||||
// next one on: the cluster's settings changed. Refused, and the old one
|
||||
// kept, when it is not valid.
|
||||
func (g *Gate) Replace(cfg Config) error {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
// The patterns first: a policy naming one must find it compiled.
|
||||
old := *g.deny.Load()
|
||||
next := *compileDeny(cfg)
|
||||
for k, v := range old {
|
||||
if _, ok := next[k]; !ok {
|
||||
next[k] = v
|
||||
}
|
||||
}
|
||||
g.deny.Store(&next)
|
||||
g.mu.Lock()
|
||||
g.cfg = cfg
|
||||
g.merged = map[string]Policy{}
|
||||
g.mu.Unlock()
|
||||
g.reload(context.Background())
|
||||
return g, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// Library is the policies in force.
|
||||
func (g *Gate) Library() Config {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
return g.cfg
|
||||
}
|
||||
|
||||
// attachmentsTTL is how stale an attachment may be: one made in the console
|
||||
@@ -92,7 +130,7 @@ func (g *Gate) reload(ctx context.Context) {
|
||||
g.attached = map[string][]string{}
|
||||
for _, a := range list {
|
||||
if _, ok := g.cfg.Policies[a.Policy]; !ok {
|
||||
g.log.Warn("a guard policy is attached that server.yaml does not define; ignored",
|
||||
g.log.Warn("a guard policy is attached that the cluster's settings do not define; ignored",
|
||||
"policy", a.Policy, "scope", a.Scope, "id", a.ScopeID)
|
||||
continue
|
||||
}
|
||||
@@ -188,7 +226,7 @@ func (g *Gate) judge(text string, r Rules) ([]finding, []detect.Span) {
|
||||
if len(r.Deny) > 0 {
|
||||
norm := detect.Normalize(text)
|
||||
for _, d := range r.Deny {
|
||||
re := g.deny[d.Regex]
|
||||
re := (*g.deny.Load())[d.Regex]
|
||||
if re == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -69,7 +69,11 @@ type Options struct {
|
||||
// NoFirewall leaves the machine's firewall as it is. Otherwise its llmgw
|
||||
// ports — FirewallPorts — answer only the networks it reaches the
|
||||
// control planes on, and FirewallAllow.
|
||||
NoFirewall bool
|
||||
NoFirewall bool
|
||||
// MirrorBase is where the mirror serves model files, http://host:8081:
|
||||
// the node is asked to reach it before it is added, since a node that
|
||||
// cannot fetches no model.
|
||||
MirrorBase string
|
||||
FirewallPorts []firewall.Port
|
||||
FirewallAllow []netip.Prefix
|
||||
|
||||
@@ -461,6 +465,9 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
|
||||
// A missing firewall is what this run is about to write.
|
||||
rep.Findings = slices.DeleteFunc(rep.Findings, func(f doctor.Finding) bool { return f.Area == "firewall" })
|
||||
}
|
||||
if f, ok := r.mirrorReach(ctx); ok {
|
||||
rep.Findings = append(rep.Findings, f)
|
||||
}
|
||||
res.Facts, res.Findings = rep.Facts, rep.Findings
|
||||
|
||||
fails := 0
|
||||
@@ -502,6 +509,37 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// mirrorReach asks the node to reach the mirror's files: a base address the
|
||||
// node cannot reach (another machine's interface, Docker's bridge, a port
|
||||
// closed to it) otherwise shows up as every model failing to download.
|
||||
func (r *Runner) mirrorReach(ctx context.Context) (doctor.Finding, bool) {
|
||||
if r.Opts.MirrorBase == "" {
|
||||
return doctor.Finding{}, false
|
||||
}
|
||||
u, err := url.Parse(r.Opts.MirrorBase)
|
||||
if err != nil || u.Hostname() == "" {
|
||||
return doctor.Finding{}, false
|
||||
}
|
||||
port := u.Port()
|
||||
if port == "" {
|
||||
port = "80"
|
||||
}
|
||||
base := strings.TrimRight(r.Opts.MirrorBase, "/") + "/"
|
||||
out, _ := r.Remote.Run(ctx, `if command -v curl >/dev/null 2>&1; then
|
||||
curl -s -o /dev/null -w '%{http_code}' --max-time 5 `+shellQuote(base)+` || true
|
||||
else
|
||||
timeout 5 bash -c `+shellQuote("</dev/tcp/"+u.Hostname()+"/"+port)+` 2>/dev/null && echo 200 || echo 000
|
||||
fi`, nil)
|
||||
code := strings.TrimSpace(lastLine(out))
|
||||
if code != "" && code != "000" {
|
||||
return doctor.Finding{Level: doctor.OK, Area: "mirror", Message: "reaches the mirror's files at " + base}, true
|
||||
}
|
||||
return doctor.Finding{Level: doctor.Fail, Area: "mirror",
|
||||
Message: "cannot reach the mirror's files at " + base + ": every model it is given would fail to download",
|
||||
Fix: "on the mirror's machine, give llmgw-mirror a --base this node reaches (systemctl cat llmgw-mirror), " +
|
||||
"and let this node through to port " + port}, true
|
||||
}
|
||||
|
||||
// suggestFabrics groups the active adapters by the fabric they report being
|
||||
// on: the subnet manager they see, or the network of their RoCE address.
|
||||
// Names are placeholders for the operator: a name is what ties one node's
|
||||
|
||||
@@ -32,6 +32,8 @@ type fakeMachine struct {
|
||||
files map[string][]byte
|
||||
scripts []string
|
||||
download error
|
||||
// mirror is what curl prints for the mirror's address: 000 unreachable.
|
||||
mirror string
|
||||
}
|
||||
|
||||
type doctorReport struct {
|
||||
@@ -64,6 +66,8 @@ func (f *fakeMachine) Run(_ context.Context, script string, stdin []byte) ([]byt
|
||||
return nil, nil
|
||||
case strings.Contains(script, "systemctl restart llmgw-agent"):
|
||||
return []byte("active\n"), nil
|
||||
case strings.Contains(script, "curl -s -o /dev/null -w"):
|
||||
return []byte(f.mirror), nil
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
@@ -389,3 +393,30 @@ func TestTheSudoGrantRemovesItselfWithinTheHour(t *testing.T) {
|
||||
t.Errorf("not shell: %v %s\n%s", err, out, inner)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeThatCannotReachTheMirrorIsNotAdded(t *testing.T) {
|
||||
m := newFake()
|
||||
m.mirror = "000"
|
||||
r, events := runner(t, m, Options{Addr: "10.0.0.9", MirrorBase: "http://172.17.0.1:8081"})
|
||||
var res Result
|
||||
err := r.check(t.Context(), &res)
|
||||
if !errors.Is(err, ErrChecks) {
|
||||
t.Fatalf("a node that cannot reach the mirror was let through: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, e := range *events {
|
||||
if e.Level == Fail && strings.Contains(e.Detail, "http://172.17.0.1:8081/") && strings.Contains(e.Fix, "--base") {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("no failure naming the address and the fix: %+v", *events)
|
||||
}
|
||||
|
||||
ok := newFake()
|
||||
ok.mirror = "404"
|
||||
r, _ = runner(t, ok, Options{Addr: "10.0.0.9", MirrorBase: "http://10.0.0.146:8081"})
|
||||
if err := r.check(t.Context(), &res); err != nil {
|
||||
t.Errorf("a node that reaches the mirror: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -208,6 +208,10 @@ func (r *Router) Handler() http.Handler {
|
||||
// So the example on the profile page names a model that is actually
|
||||
// running, rather than a placeholder that gets sent as-is.
|
||||
pages.ServedModels(r.exampleModels)
|
||||
pages.SetModelFacts(func(name string) web.ModelFacts {
|
||||
_, m := r.resolve(name)
|
||||
return web.ModelFacts{ContextLength: m.ContextLength, MaxOutput: m.MaxOutput, Tools: m.Capabilities.Tools}
|
||||
})
|
||||
pages.SetPersonalOff(r.personalOff)
|
||||
pages.SetLlamaCppVersion(r.llamaVersion)
|
||||
pages.SetPersonalDefaults(r.personalDefaults)
|
||||
@@ -235,7 +239,7 @@ func (r *Router) Handler() http.Handler {
|
||||
return names, p.NoStore && p.Mode == "enforce"
|
||||
})
|
||||
}
|
||||
if r.store != nil && len(r.guardLibrary.Policies) > 0 {
|
||||
if r.store != nil && r.guardLibrary != nil {
|
||||
pages.RegisterAdminGuard(mux, r.store, r.guardLibrary)
|
||||
}
|
||||
pages.Register(mux)
|
||||
|
||||
@@ -108,7 +108,7 @@ type Router struct {
|
||||
// guard judges requests and answers; nil judges nothing. guardLibrary
|
||||
// is the policies it was built from, for the pages that show them.
|
||||
guard guard.Guard
|
||||
guardLibrary guard.Config
|
||||
guardLibrary func() guard.Config
|
||||
// oidc is sign-in through an identity provider, and scim provisioning
|
||||
// by it.
|
||||
oidc config.OIDC
|
||||
@@ -187,7 +187,7 @@ func (r *Router) SetMirror(m web.MirrorAdmin) { r.mirror = m }
|
||||
func (r *Router) SetGuard(g guard.Guard) { r.guard = g }
|
||||
|
||||
// SetGuardLibrary is the policies the guard was built from, for the pages.
|
||||
func (r *Router) SetGuardLibrary(c guard.Config) { r.guardLibrary = c }
|
||||
func (r *Router) SetGuardLibrary(c func() guard.Config) { r.guardLibrary = c }
|
||||
|
||||
// SetRegistry gives the mirror page somewhere to look models up.
|
||||
func (r *Router) SetRegistry(reg web.Registry) { r.registry = reg }
|
||||
|
||||
@@ -126,15 +126,61 @@ type Doc struct {
|
||||
doc yaml.Node
|
||||
// Changed counts edits not yet saved.
|
||||
Changed int
|
||||
|
||||
// cluster holds the keys that are not the machine's own, when the
|
||||
// cluster keeps them elsewhere (the store); isLocal says which are.
|
||||
cluster *Doc
|
||||
isLocal func([]string) bool
|
||||
}
|
||||
|
||||
// Layer has the keys isLocal does not claim read and written in cluster
|
||||
// instead of this document: one screen over a machine's file and the
|
||||
// cluster's settings.
|
||||
func (d *Doc) Layer(cluster *Doc, isLocal func([]string) bool) {
|
||||
d.cluster, d.isLocal = cluster, isLocal
|
||||
}
|
||||
|
||||
// Cluster is the layered cluster document, nil when there is none.
|
||||
func (d *Doc) Cluster() *Doc { return d.cluster }
|
||||
|
||||
// Pending counts edits not yet saved, in this document and its layer.
|
||||
func (d *Doc) Pending() int {
|
||||
if d.cluster != nil {
|
||||
return d.Changed + d.cluster.Changed
|
||||
}
|
||||
return d.Changed
|
||||
}
|
||||
|
||||
// at is the document a path is kept in.
|
||||
func (d *Doc) at(path []string) *Doc {
|
||||
if d.cluster != nil && len(path) > 0 && !d.isLocal(path) {
|
||||
return d.cluster
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// Open reads a file; a missing one is an empty document.
|
||||
func Open(path string) (*Doc, error) {
|
||||
d := &Doc{Path: path}
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return nil, err
|
||||
}
|
||||
return parseDoc(path, b)
|
||||
}
|
||||
|
||||
// FromBytes reads a document held somewhere other than a file: the
|
||||
// cluster's settings, kept in the store. It has no Path, so it is not Saved.
|
||||
func FromBytes(name string, b []byte) (*Doc, error) {
|
||||
d, err := parseDoc(name, b)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
d.Path = ""
|
||||
return d, nil
|
||||
}
|
||||
|
||||
func parseDoc(path string, b []byte) (*Doc, error) {
|
||||
d := &Doc{Path: path}
|
||||
if len(bytes.TrimSpace(b)) > 0 {
|
||||
if err := yaml.Unmarshal(b, &d.doc); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", path, err)
|
||||
@@ -163,6 +209,9 @@ func (d *Doc) Bytes() ([]byte, error) {
|
||||
|
||||
// Node is the node at a path of keys and list indexes, or nil.
|
||||
func (d *Doc) Node(path []string) *yaml.Node {
|
||||
if t := d.at(path); t != d {
|
||||
return t.Node(path)
|
||||
}
|
||||
n := d.root
|
||||
for _, p := range path {
|
||||
n = child(n, p)
|
||||
@@ -209,6 +258,9 @@ func (d *Doc) ensure(path []string) *yaml.Node {
|
||||
// Set writes one key's value: a scalar, a list of lines or a map of
|
||||
// "key = value" lines, by its kind. It checks the value first.
|
||||
func (d *Doc) Set(path []string, k Kind, value string) error {
|
||||
if t := d.at(path); t != d {
|
||||
return t.Set(path, k, value)
|
||||
}
|
||||
if len(path) == 0 {
|
||||
return fmt.Errorf("no key")
|
||||
}
|
||||
@@ -216,6 +268,42 @@ func (d *Doc) Set(path []string, k Kind, value string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return d.place(path, n)
|
||||
}
|
||||
|
||||
// SetYAML writes one key's value given as YAML: a scalar, [a, b], {k: v},
|
||||
// or a whole block.
|
||||
func (d *Doc) SetYAML(path []string, value string) error {
|
||||
if t := d.at(path); t != d {
|
||||
return t.SetYAML(path, value)
|
||||
}
|
||||
if len(path) == 0 {
|
||||
return fmt.Errorf("no key")
|
||||
}
|
||||
var doc yaml.Node
|
||||
if err := yaml.Unmarshal([]byte(value), &doc); err != nil {
|
||||
return fmt.Errorf("%q is not YAML: %w", value, err)
|
||||
}
|
||||
n := &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!null", Value: ""}
|
||||
if len(doc.Content) > 0 {
|
||||
n = doc.Content[0]
|
||||
}
|
||||
return d.place(path, n)
|
||||
}
|
||||
|
||||
// SetNode writes a node, as read from another document, at path.
|
||||
func (d *Doc) SetNode(path []string, n *yaml.Node) error {
|
||||
if t := d.at(path); t != d {
|
||||
return t.SetNode(path, n)
|
||||
}
|
||||
if len(path) == 0 {
|
||||
return fmt.Errorf("no key")
|
||||
}
|
||||
c := *n
|
||||
return d.place(path, &c)
|
||||
}
|
||||
|
||||
func (d *Doc) place(path []string, n *yaml.Node) error {
|
||||
parent := d.ensure(path[:len(path)-1])
|
||||
if parent == nil {
|
||||
return fmt.Errorf("%s is not a block", strings.Join(path[:len(path)-1], "."))
|
||||
@@ -237,6 +325,10 @@ func (d *Doc) Set(path []string, k Kind, value string) error {
|
||||
|
||||
// Unset removes a key, so its default applies.
|
||||
func (d *Doc) Unset(path []string) {
|
||||
if t := d.at(path); t != d {
|
||||
t.Unset(path)
|
||||
return
|
||||
}
|
||||
if len(path) == 0 {
|
||||
return
|
||||
}
|
||||
@@ -264,6 +356,9 @@ func (d *Doc) Unset(path []string) {
|
||||
|
||||
// Append adds an empty block to the list at path, and returns its index.
|
||||
func (d *Doc) Append(path []string) int {
|
||||
if t := d.at(path); t != d {
|
||||
return t.Append(path)
|
||||
}
|
||||
parent := d.ensure(path[:len(path)-1])
|
||||
key := path[len(path)-1]
|
||||
seq := child(parent, key)
|
||||
@@ -283,6 +378,9 @@ func (d *Doc) Append(path []string) int {
|
||||
|
||||
// AddEntry adds an empty block under a name to the map at path.
|
||||
func (d *Doc) AddEntry(path []string, name string) error {
|
||||
if t := d.at(path); t != d {
|
||||
return t.AddEntry(path, name)
|
||||
}
|
||||
m := d.ensure(path)
|
||||
if m == nil || m.Kind != yaml.MappingNode {
|
||||
return fmt.Errorf("%s is not a map", strings.Join(path, "."))
|
||||
|
||||
@@ -58,6 +58,8 @@ var copyOrder = []string{
|
||||
// Or a removed node's certificate would work again in the new store.
|
||||
"revoked_nodes",
|
||||
"guard_attachments",
|
||||
// Or the cluster would start the new store on server.yaml's settings again.
|
||||
"cluster_settings",
|
||||
// Who changed what is not lost by moving the store.
|
||||
"audit_events",
|
||||
"key_ttl",
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- The cluster's settings: everything in server.yaml that is not one
|
||||
-- machine's own, as a versioned YAML document every control plane follows.
|
||||
CREATE TABLE cluster_settings (
|
||||
version BIGINT PRIMARY KEY,
|
||||
body TEXT NOT NULL,
|
||||
note TEXT NOT NULL DEFAULT '',
|
||||
created_at BIGINT NOT NULL,
|
||||
created_by TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
@@ -0,0 +1,13 @@
|
||||
-- The cluster's settings: everything in server.yaml that is not one
|
||||
-- machine's own (listeners, certificates, the store's address), as a YAML
|
||||
-- document every control plane reads and follows. Each change is a new row,
|
||||
-- so what was in force before is there to compare with and go back to, and
|
||||
-- two edits made at once cannot both win: the second is refused for naming
|
||||
-- a version that is no longer the newest.
|
||||
CREATE TABLE cluster_settings (
|
||||
version INTEGER PRIMARY KEY,
|
||||
body TEXT NOT NULL,
|
||||
note TEXT NOT NULL DEFAULT '',
|
||||
created_at INTEGER NOT NULL,
|
||||
created_by TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
@@ -0,0 +1,98 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrSettingsChanged is a change made against a version of the cluster's
|
||||
// settings that is no longer the newest: somebody else changed them first.
|
||||
var ErrSettingsChanged = errors.New("the cluster's settings were changed since they were read; read them again")
|
||||
|
||||
// SettingsVersion is one version of the cluster's settings.
|
||||
type SettingsVersion struct {
|
||||
Version int64
|
||||
Body string
|
||||
Note string
|
||||
At time.Time
|
||||
By string
|
||||
}
|
||||
|
||||
// ClusterSettings is the newest version, Version 0 when there is none yet.
|
||||
func (s *DB) ClusterSettings(ctx context.Context) (SettingsVersion, error) {
|
||||
return s.settingsWhere(ctx, `version = (SELECT MAX(version) FROM cluster_settings)`)
|
||||
}
|
||||
|
||||
// ClusterSettingsAt is one version, Version 0 when there is no such version.
|
||||
func (s *DB) ClusterSettingsAt(ctx context.Context, version int64) (SettingsVersion, error) {
|
||||
return s.settingsWhere(ctx, `version = ?`, version)
|
||||
}
|
||||
|
||||
func (s *DB) settingsWhere(ctx context.Context, where string, args ...any) (SettingsVersion, error) {
|
||||
var v SettingsVersion
|
||||
var at int64
|
||||
err := s.db.QueryRowContext(ctx,
|
||||
`SELECT version, body, note, created_at, created_by FROM cluster_settings WHERE `+where, args...).
|
||||
Scan(&v.Version, &v.Body, &v.Note, &at, &v.By)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return SettingsVersion{}, nil
|
||||
}
|
||||
v.At = time.UnixMilli(at)
|
||||
return v, err
|
||||
}
|
||||
|
||||
// ClusterSettingsVersion is the newest version's number, 0 when there is
|
||||
// none: what every control plane asks every few seconds.
|
||||
func (s *DB) ClusterSettingsVersion(ctx context.Context) (int64, error) {
|
||||
var v sql.NullInt64
|
||||
err := s.db.QueryRowContext(ctx, `SELECT MAX(version) FROM cluster_settings`).Scan(&v)
|
||||
return v.Int64, err
|
||||
}
|
||||
|
||||
// PutClusterSettings records a new version made from base, the version the
|
||||
// change was read from, and returns its number. Refused with
|
||||
// ErrSettingsChanged when base is not the newest.
|
||||
func (s *DB) PutClusterSettings(ctx context.Context, base int64, body, by, note string) (int64, error) {
|
||||
cur, err := s.ClusterSettingsVersion(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if cur != base {
|
||||
return 0, ErrSettingsChanged
|
||||
}
|
||||
// The key settles a race the read above cannot: of two writers from the
|
||||
// same base, one inserts base+1 and the other is refused.
|
||||
_, err = s.db.ExecContext(ctx,
|
||||
`INSERT INTO cluster_settings (version, body, note, created_at, created_by) VALUES (?, ?, ?, ?, ?)`,
|
||||
base+1, body, note, time.Now().UnixMilli(), by)
|
||||
if isUniqueViolation(err) {
|
||||
return 0, ErrSettingsChanged
|
||||
}
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return base + 1, nil
|
||||
}
|
||||
|
||||
// ClusterSettingsHistory is the newest versions first, at most limit.
|
||||
func (s *DB) ClusterSettingsHistory(ctx context.Context, limit int) ([]SettingsVersion, error) {
|
||||
rows, err := s.db.QueryContext(ctx,
|
||||
`SELECT version, body, note, created_at, created_by FROM cluster_settings ORDER BY version DESC LIMIT ?`, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []SettingsVersion
|
||||
for rows.Next() {
|
||||
var v SettingsVersion
|
||||
var at int64
|
||||
if err := rows.Scan(&v.Version, &v.Body, &v.Note, &at, &v.By); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v.At = time.UnixMilli(at)
|
||||
out = append(out, v)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSettingsAreVersionedAndTwoEditsCannotBothWin(t *testing.T) {
|
||||
s := capacityStore(t)
|
||||
ctx := t.Context()
|
||||
if v, err := s.ClusterSettings(ctx); err != nil || v.Version != 0 {
|
||||
t.Fatalf("empty store: %+v %v", v, err)
|
||||
}
|
||||
v1, err := s.PutClusterSettings(ctx, 0, "router: {}\n", "import", "from server.yaml")
|
||||
if err != nil || v1 != 1 {
|
||||
t.Fatalf("first: %d %v", v1, err)
|
||||
}
|
||||
if _, err := s.PutClusterSettings(ctx, 0, "x: 1\n", "late", ""); !errors.Is(err, ErrSettingsChanged) {
|
||||
t.Fatalf("a change from an old version: %v", err)
|
||||
}
|
||||
if _, err := s.PutClusterSettings(ctx, 1, "guard: {}\n", "ilker", "guard"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cur, _ := s.ClusterSettings(ctx)
|
||||
if cur.Version != 2 || cur.Body != "guard: {}\n" || cur.By != "ilker" {
|
||||
t.Errorf("newest: %+v", cur)
|
||||
}
|
||||
if n, _ := s.ClusterSettingsVersion(ctx); n != 2 {
|
||||
t.Errorf("version %d", n)
|
||||
}
|
||||
old, _ := s.ClusterSettingsAt(ctx, 1)
|
||||
if old.Body != "router: {}\n" || old.Note != "from server.yaml" {
|
||||
t.Errorf("version 1: %+v", old)
|
||||
}
|
||||
h, _ := s.ClusterSettingsHistory(ctx, 10)
|
||||
if len(h) != 2 || h[0].Version != 2 {
|
||||
t.Errorf("history: %+v", h)
|
||||
}
|
||||
}
|
||||
@@ -77,6 +77,8 @@ type AccountPages struct {
|
||||
// field in that snippet is real, so the one placeholder left in it was
|
||||
// copied verbatim and sent as the literal string MODEL.
|
||||
served func() []string
|
||||
// modelFacts is what the catalog says of a model, for an editor.
|
||||
modelFacts func(model string) ModelFacts
|
||||
// hasMirror is set when /admin/mirror is registered, so the other admin
|
||||
// pages can link to it and only then.
|
||||
hasMirror bool
|
||||
@@ -132,6 +134,9 @@ type AccountPages struct {
|
||||
// example falls back to a placeholder, which is what it always was.
|
||||
func (p *AccountPages) ServedModels(f func() []string) { p.served = f }
|
||||
|
||||
// SetModelFacts supplies what an editor's configuration says of a model.
|
||||
func (p *AccountPages) SetModelFacts(f func(model string) ModelFacts) { p.modelFacts = f }
|
||||
|
||||
// exampleModel picks the model the copy-pasteable example should name.
|
||||
//
|
||||
// A real model over a simulated one is not expressible here — this package does
|
||||
|
||||
@@ -49,17 +49,18 @@ var guardWindows = map[string]time.Duration{"24h": 24 * time.Hour, "7d": 7 * 24
|
||||
|
||||
// RegisterAdminGuard mounts /admin/guard: what the guardrails did, where, by
|
||||
// which policy — names and counts, never text.
|
||||
func (p *AccountPages) RegisterAdminGuard(mux *http.ServeMux, s GuardAdminSource, library guard.Config) {
|
||||
func (p *AccountPages) RegisterAdminGuard(mux *http.ServeMux, s GuardAdminSource, library func() guard.Config) {
|
||||
p.hasGuard = true
|
||||
mux.HandleFunc("GET /admin/guard", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
ctx, l := r.Context(), langOf(r)
|
||||
lib := library()
|
||||
d := guardPageData{Nav: p.navFor(r, admin, "guard"), Version: p.version, Window: r.URL.Query().Get("window"),
|
||||
Windows: []string{"24h", "7d", "30d"}, Default: library.Default}
|
||||
Windows: []string{"24h", "7d", "30d"}, Default: lib.Default}
|
||||
since, ok := guardWindows[d.Window]
|
||||
if !ok {
|
||||
d.Window, since = "7d", guardWindows["7d"]
|
||||
}
|
||||
for name := range library.Policies {
|
||||
for name := range lib.Policies {
|
||||
d.Policies = append(d.Policies, name)
|
||||
}
|
||||
slices.Sort(d.Policies)
|
||||
|
||||
@@ -591,8 +591,9 @@ func TestHowAModelRunsIsChangedWhereItsFileIsShown(t *testing.T) {
|
||||
if !strings.Contains(body, `<option value="always" selected>always on</option>`) {
|
||||
t.Error("the running model's mode is not shown as chosen")
|
||||
}
|
||||
// Declared in server.yaml: shown, not offered, since it would revert.
|
||||
if !strings.Contains(body, `title="set in server.yaml, which is read again on every start">on demand · yaml</span>`) {
|
||||
// Declared in the cluster's settings: shown, not offered, since it is
|
||||
// changed there.
|
||||
if !strings.Contains(body, `title="set in the cluster's settings (llmgw ctl config)">on demand · yaml</span>`) {
|
||||
t.Error("a declared model should say where its mode is set")
|
||||
}
|
||||
if strings.Count(body, `name="mode" class="mode"`) != 1 {
|
||||
|
||||
+60
-12
@@ -45,10 +45,25 @@ type connectField struct{ Label, Value string }
|
||||
// unquoted: the names this gateway serves, and nothing a shell reads twice.
|
||||
var safeModel = regexp.MustCompile(`^[A-Za-z0-9._:/@+-]{1,200}$`)
|
||||
|
||||
// ModelFacts is what an editor's configuration needs to know of a model: how
|
||||
// much one request may hold, and whether its tool calls come back as calls.
|
||||
type ModelFacts struct {
|
||||
ContextLength int
|
||||
// MaxOutput is the longest answer the gateway allows; 0 is no limit.
|
||||
MaxOutput int
|
||||
Tools bool
|
||||
}
|
||||
|
||||
// continueYAML is a Continue assistant pointing at the gateway: a file of its
|
||||
// own under ~/.continue/assistants, so the person's config.yaml is not touched.
|
||||
func continueYAML(base, model, key string) string {
|
||||
return fmt.Sprintf(`# Written by llmgw for Continue: the gateway at %[1]s.
|
||||
//
|
||||
// What it says of the model is what the catalog says. Continue assumed a
|
||||
// context far larger than a CPU replica's, and every request over it was
|
||||
// refused; and told a model uses tools when its calls come back as text,
|
||||
// Continue's agent stalls on them rather than acting.
|
||||
func continueYAML(base, model, key string, f ModelFacts) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, `# Written by llmgw for Continue: the gateway at %[1]s.
|
||||
name: llmgw
|
||||
version: 1.0.0
|
||||
schema: v1
|
||||
@@ -62,11 +77,25 @@ models:
|
||||
- chat
|
||||
- edit
|
||||
- apply
|
||||
capabilities:
|
||||
- tool_use
|
||||
# For completion as you type, add a small, fast model here with the role
|
||||
# autocomplete; a large one makes every keystroke wait for it.
|
||||
`, base, model, key)
|
||||
if f.Tools {
|
||||
b.WriteString(" capabilities:\n - tool_use\n")
|
||||
}
|
||||
if f.ContextLength > 0 {
|
||||
fmt.Fprintf(&b, ` # What the gateway lets one request hold; Continue trims to fit.
|
||||
defaultCompletionOptions:
|
||||
contextLength: %d
|
||||
maxTokens: %d
|
||||
`, f.ContextLength, maxTokensFor(f))
|
||||
}
|
||||
if !f.Tools {
|
||||
b.WriteString("# This model's catalog entry does not say it makes tool calls, so Continue\n" +
|
||||
"# chats and edits with it but its agent cannot act: choose one that does.\n")
|
||||
}
|
||||
b.WriteString(`# For completion as you type, add a small, fast model here with the role
|
||||
# autocomplete; a large one makes every keystroke wait for it.
|
||||
`)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// codexTOML is the provider and profile Codex needs, and nothing else of its
|
||||
@@ -87,13 +116,13 @@ model = "%[2]s"
|
||||
}
|
||||
|
||||
// connectFor builds what the page shows for a tool, in English.
|
||||
func connectFor(tool, base, model, key string) connectView {
|
||||
return connectIn(english, tool, base, model, key)
|
||||
func connectFor(tool, base, model, key string, f ModelFacts) connectView {
|
||||
return connectIn(english, tool, base, model, key, f)
|
||||
}
|
||||
|
||||
// connectIn builds what the page shows for a tool, its sentences in l. The
|
||||
// files, commands and paths are the same in every language.
|
||||
func connectIn(l *Lang, tool, base, model, key string) connectView {
|
||||
func connectIn(l *Lang, tool, base, model, key string, f ModelFacts) connectView {
|
||||
v := connectView{Tool: tool, Model: model, Key: key}
|
||||
for _, t := range editorTools {
|
||||
if t.Key == tool {
|
||||
@@ -104,7 +133,7 @@ func connectIn(l *Lang, tool, base, model, key string) connectView {
|
||||
switch tool {
|
||||
case "continue":
|
||||
v.Files = []connectFile{{Path: "~/.continue/assistants/llmgw.yaml", Name: "llmgw.yaml",
|
||||
Body: continueYAML(base, model, key)}}
|
||||
Body: continueYAML(base, model, key, f)}}
|
||||
v.Command = fmt.Sprintf("curl -fsSL '%s' | LLMGW_KEY='%s' sh", script, key)
|
||||
v.Run = l.T("connect.run.continue")
|
||||
case "codex":
|
||||
@@ -173,7 +202,7 @@ func (p *AccountPages) connect(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
p.log.Info("a user connected an editor", "user", user.ID, "tool", tool, "key", k.ID)
|
||||
c := connectIn(l, tool, baseURL(r), model, secret)
|
||||
c := connectIn(l, tool, baseURL(r), model, secret, p.facts(model))
|
||||
p.render(w, r, sess, user, profileData{Connect: &c})
|
||||
}
|
||||
|
||||
@@ -210,7 +239,7 @@ cat > "$file" <<'LLMGW'
|
||||
tmp="$file.tmp"
|
||||
sed "s|__LLMGW_KEY__|$LLMGW_KEY|" "$file" > "$tmp" && mv "$tmp" "$file"
|
||||
echo "Wrote $file. In Continue, choose the llmgw assistant."
|
||||
`, base, continueYAML(base, model, "__LLMGW_KEY__"))
|
||||
`, base, continueYAML(base, model, "__LLMGW_KEY__", p.facts(model)))
|
||||
case "codex":
|
||||
body = fmt.Sprintf(`#!/bin/sh
|
||||
# Connects Codex to the llmgw gateway at %[1]s.
|
||||
@@ -242,3 +271,22 @@ echo "Run: LLMGW_API_KEY=<your key> codex --profile llmgw"
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
_, _ = w.Write([]byte(body))
|
||||
}
|
||||
|
||||
// facts is what the catalog says of a model, when there is one to ask.
|
||||
func (p *AccountPages) facts(model string) ModelFacts {
|
||||
if p.modelFacts == nil {
|
||||
return ModelFacts{}
|
||||
}
|
||||
return p.modelFacts(model)
|
||||
}
|
||||
|
||||
// maxTokensFor is the answer length Continue asks for: a quarter of the
|
||||
// context, at most 4096, and never over what the gateway allows — asked for
|
||||
// more, it refuses the request.
|
||||
func maxTokensFor(f ModelFacts) int {
|
||||
n := min(4096, max(256, f.ContextLength/4))
|
||||
if f.MaxOutput > 0 {
|
||||
n = min(n, f.MaxOutput)
|
||||
}
|
||||
return n
|
||||
}
|
||||
@@ -11,7 +11,7 @@ import (
|
||||
)
|
||||
|
||||
func TestEachToolGetsItsConfigurationWithTheKey(t *testing.T) {
|
||||
c := connectFor("continue", "https://gw.example", "qwen2.5-coder-7b", "sk-llmgw-abc")
|
||||
c := connectFor("continue", "https://gw.example", "qwen2.5-coder-7b", "sk-llmgw-abc", ModelFacts{})
|
||||
if !strings.Contains(c.Files[0].Body, "apiBase: https://gw.example/v1") || !strings.Contains(c.Files[0].Body, "apiKey: sk-llmgw-abc") ||
|
||||
c.Files[0].Path != "~/.continue/assistants/llmgw.yaml" {
|
||||
t.Errorf("continue = %+v", c.Files)
|
||||
@@ -21,11 +21,11 @@ func TestEachToolGetsItsConfigurationWithTheKey(t *testing.T) {
|
||||
if strings.Contains(strings.Split(c.Command, "|")[0], "sk-llmgw-abc") {
|
||||
t.Errorf("the key is in the URL: %s", c.Command)
|
||||
}
|
||||
x := connectFor("codex", "https://gw.example", "qwen2.5-coder-7b", "sk-llmgw-abc")
|
||||
x := connectFor("codex", "https://gw.example", "qwen2.5-coder-7b", "sk-llmgw-abc", ModelFacts{})
|
||||
if strings.Contains(x.Files[0].Body, "sk-llmgw-abc") || !strings.Contains(x.Files[0].Body, `env_key = "LLMGW_API_KEY"`) {
|
||||
t.Errorf("codex writes the key into its file: %s", x.Files[0].Body)
|
||||
}
|
||||
cl := connectFor("cline", "https://gw.example", "m", "sk-llmgw-abc")
|
||||
cl := connectFor("cline", "https://gw.example", "m", "sk-llmgw-abc", ModelFacts{})
|
||||
if len(cl.Fields) != 4 || cl.Fields[1].Value != "https://gw.example/v1" || cl.Command != "" {
|
||||
t.Errorf("cline = %+v", cl)
|
||||
}
|
||||
@@ -115,3 +115,31 @@ func TestASetupScriptRefusesANameAShellWouldRead(t *testing.T) {
|
||||
t.Errorf("%d", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContinueIsToldTheModelsContextAndWhetherItCanUseTools(t *testing.T) {
|
||||
// Told a context far larger than the gateway allows, Continue sends
|
||||
// prompts that are refused; told a model uses tools when its calls come
|
||||
// back as text, its agent stalls.
|
||||
agent := continueYAML("https://gw.example", "coder-30b", "k", ModelFacts{ContextLength: 32768, Tools: true})
|
||||
for _, want := range []string{"tool_use", "contextLength: 32768", "maxTokens: 4096"} {
|
||||
if !strings.Contains(agent, want) {
|
||||
t.Errorf("%q missing:\n%s", want, agent)
|
||||
}
|
||||
}
|
||||
chat := continueYAML("https://gw.example", "coder", "k", ModelFacts{ContextLength: 4096})
|
||||
if strings.Contains(chat, "tool_use") || !strings.Contains(chat, "contextLength: 4096") ||
|
||||
!strings.Contains(chat, "maxTokens: 1024") || !strings.Contains(chat, "its agent cannot act") {
|
||||
t.Errorf("a model without tools:\n%s", chat)
|
||||
}
|
||||
if unknown := continueYAML("https://gw.example", "m", "k", ModelFacts{}); strings.Contains(unknown, "contextLength") {
|
||||
t.Errorf("a context nobody knows was written down:\n%s", unknown)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContinueNeverAsksForMoreThanTheGatewayAllows(t *testing.T) {
|
||||
// Asked for more than max_output, the gateway refuses the request.
|
||||
y := continueYAML("https://gw.example", "m", "k", ModelFacts{ContextLength: 32768, MaxOutput: 2048, Tools: true})
|
||||
if !strings.Contains(y, "maxTokens: 2048") {
|
||||
t.Errorf("not capped at max_output:\n%s", y)
|
||||
}
|
||||
}
|
||||
@@ -138,14 +138,15 @@ func TestTheGuardrailsPageSpeaksTurkishWhenAsked(t *testing.T) {
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
at := time.Date(2026, 10, 4, 9, 5, 7, 0, time.UTC)
|
||||
p.RegisterAdminGuard(mux, guardSource{at: at}, guard.Config{
|
||||
Default: []string{"base"}, Policies: map[string]guard.Policy{"base": {}, "finance": {}}})
|
||||
p.RegisterAdminGuard(mux, guardSource{at: at}, func() guard.Config {
|
||||
return guard.Config{Default: []string{"base"}, Policies: map[string]guard.Policy{"base": {}, "finance": {}}}
|
||||
})
|
||||
|
||||
body := peopleInTurkish(t, mux, http.MethodGet, "/admin/guard", nil)
|
||||
for _, want := range []string{
|
||||
"<title>Koruma kuralları · llmgw</title>",
|
||||
"<h2>Politikalar</h2>",
|
||||
"<code>server.yaml</code> içinde tanımlı:",
|
||||
"Cluster ayarlarında tanımlı (<code>llmgw ctl config</code>):",
|
||||
"<td>proje</td><td>Acme/fin/payroll</td>", // the scope's kind, in Turkish
|
||||
"<td>kişisel</td><td>maskelendi</td>",
|
||||
`<td class="num">1.500</td>`, // a count written the Turkish way
|
||||
|
||||
@@ -171,18 +171,18 @@ func TestTheProfilePageShowsTurkishNumbersAndDates(t *testing.T) {
|
||||
|
||||
func TestTheEditorSetupExplainsItselfInTurkish(t *testing.T) {
|
||||
tr := langByCode("tr")
|
||||
c := connectIn(tr, "cline", "https://gw.example", "m", "sk-llmgw-abc")
|
||||
c := connectIn(tr, "cline", "https://gw.example", "m", "sk-llmgw-abc", ModelFacts{})
|
||||
if c.Fields[2].Label != "API anahtarı (API Key)" || c.Fields[0].Value != "OpenAI Compatible" ||
|
||||
!strings.Contains(c.Run, "dişli simgesine") {
|
||||
t.Errorf("cline in Turkish = %+v", c)
|
||||
}
|
||||
x := connectIn(tr, "codex", "https://gw.example", "m", "sk-llmgw-abc")
|
||||
x := connectIn(tr, "codex", "https://gw.example", "m", "sk-llmgw-abc", ModelFacts{})
|
||||
// What is typed or written to a file stays what the tool reads.
|
||||
if x.Run != "LLMGW_API_KEY='sk-llmgw-abc' codex --profile llmgw" || !strings.Contains(x.Files[0].Body, "[profiles.llmgw]") ||
|
||||
!strings.Contains(x.Notes[0], "~/.codex/config.toml") {
|
||||
t.Errorf("codex in Turkish = %+v", x)
|
||||
}
|
||||
if en := connectFor("continue", "https://gw.example", "m", "k"); en.Run != `In Continue, choose the "llmgw" assistant at the top of its panel.` {
|
||||
if en := connectFor("continue", "https://gw.example", "m", "k", ModelFacts{}); en.Run != `In Continue, choose the "llmgw" assistant at the top of its panel.` {
|
||||
t.Errorf("English Continue: %q", en.Run)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"guard.title": "Guardrails",
|
||||
"guard.sub": "What the policies did, where, and by which rule — names and counts, never what was written.",
|
||||
"guard.policies": "Policies",
|
||||
"guard.defined_in": "Defined in <code>server.yaml</code>:",
|
||||
"guard.defined_in": "Defined in the cluster's settings (<code>llmgw ctl config</code>):",
|
||||
"guard.applied": "Applied to every request:",
|
||||
"guard.col.scope": "Scope",
|
||||
"guard.col.who": "Who",
|
||||
@@ -13,7 +13,7 @@
|
||||
"guard.scope.user": "user",
|
||||
"guard.scope.key": "key",
|
||||
"guard.attached_nowhere": "Attached nowhere else. <code>llmgw ctl guard attach project:ORG/TEAM/PROJECT POLICY</code> attaches one.",
|
||||
"guard.no_policies": "No policies: add a <code>guard:</code> section to <code>server.yaml</code> (docs/guardrails.md).",
|
||||
"guard.no_policies": "No policies: add them under <code>guard</code> in the cluster's settings, <code>llmgw ctl config edit</code> (docs/guardrails.md).",
|
||||
|
||||
"guard.what_they_did": "What they did",
|
||||
"guard.over_last": "Over the last",
|
||||
|
||||
@@ -28,7 +28,7 @@
|
||||
"mirror.percent_of": "%d%% of %s",
|
||||
"mirror.so_far": "%s so far",
|
||||
"mirror.use.as": "as %s",
|
||||
"mirror.use.yaml_title": "set in server.yaml, which is read again on every start",
|
||||
"mirror.use.yaml_title": "set in the cluster's settings (llmgw ctl config)",
|
||||
"mirror.use.yaml": "%s · yaml",
|
||||
"mirror.mode.always": "always on",
|
||||
"mirror.mode.ondemand": "on demand",
|
||||
@@ -183,7 +183,7 @@
|
||||
"mirror.adapters.adapter_file": "Adapter file",
|
||||
"mirror.adapters.attach": "Attach",
|
||||
"mirror.adapters.no_choice": "No held adapter file for this model's architecture. Fetch one from Hugging Face first.",
|
||||
"mirror.detail.declared": "Set in server.yaml, which is read again on every start: change its replicas there.",
|
||||
"mirror.detail.declared": "Set in the cluster's settings: change its replicas there, with llmgw ctl config edit.",
|
||||
"mirror.form.min": "Always running",
|
||||
"mirror.form.max": "Up to, under load",
|
||||
"mirror.form.class": "Class",
|
||||
@@ -214,7 +214,7 @@
|
||||
"mirror.err.replicas_whole": "replicas are whole numbers",
|
||||
"mirror.err.min_over_max": "always running (%d) cannot be more than the most under load (%d)",
|
||||
"mirror.err.at_most": "at most %d replicas",
|
||||
"mirror.err.declared": "%s is set in server.yaml; change it there",
|
||||
"mirror.err.declared": "%s is set in the cluster's settings; change it there (llmgw ctl config edit)",
|
||||
"mirror.err.no_model": "no model %q",
|
||||
"mirror.err.nothing_fits": "nothing more fits",
|
||||
"mirror.err.no_room": "%s can always run at most %d now (%d running, room for %d more): %s",
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"guard.title": "Koruma kuralları",
|
||||
"guard.sub": "Politikaların ne yaptığı, nerede ve hangi kuralla — yalnızca adlar ve sayılar, yazılanlar asla.",
|
||||
"guard.policies": "Politikalar",
|
||||
"guard.defined_in": "<code>server.yaml</code> içinde tanımlı:",
|
||||
"guard.defined_in": "Cluster ayarlarında tanımlı (<code>llmgw ctl config</code>):",
|
||||
"guard.applied": "Her isteğe uygulanan:",
|
||||
"guard.col.scope": "Kapsam",
|
||||
"guard.col.who": "Kim",
|
||||
@@ -13,7 +13,7 @@
|
||||
"guard.scope.user": "kişi",
|
||||
"guard.scope.key": "anahtar",
|
||||
"guard.attached_nowhere": "Başka hiçbir yere bağlı değil. Bağlamak için: <code>llmgw ctl guard attach project:ORG/TEAM/PROJECT POLICY</code>",
|
||||
"guard.no_policies": "Politika yok: <code>server.yaml</code> dosyasına bir <code>guard:</code> bölümü ekleyin (docs/guardrails.md).",
|
||||
"guard.no_policies": "Politika yok: cluster ayarlarında <code>guard</code> altına ekleyin, <code>llmgw ctl config edit</code> (docs/guardrails.md).",
|
||||
|
||||
"guard.what_they_did": "Ne yaptılar",
|
||||
"guard.over_last": "Son",
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
"mirror.percent_of": "%[2]s içinden %%%[1]d",
|
||||
"mirror.so_far": "şu ana dek %s",
|
||||
"mirror.use.as": "%s adıyla",
|
||||
"mirror.use.yaml_title": "server.yaml içinde tanımlı; bu dosya her başlangıçta yeniden okunur",
|
||||
"mirror.use.yaml_title": "cluster ayarlarında tanımlı (llmgw ctl config)",
|
||||
"mirror.use.yaml": "%s · yaml",
|
||||
"mirror.mode.always": "sürekli açık",
|
||||
"mirror.mode.ondemand": "istek üzerine",
|
||||
@@ -178,7 +178,7 @@
|
||||
"mirror.adapters.adapter_file": "Adaptör dosyası",
|
||||
"mirror.adapters.attach": "Ekle",
|
||||
"mirror.adapters.no_choice": "Aynada bu modelin mimarisi için adaptör dosyası yok. Önce Hugging Face'ten bir tane indirin.",
|
||||
"mirror.detail.declared": "server.yaml içinde tanımlı; bu dosya her başlangıçta yeniden okunur: replikalarını orada değiştirin.",
|
||||
"mirror.detail.declared": "cluster ayarlarında tanımlı: replikalarını orada değiştirin, llmgw ctl config edit ile.",
|
||||
"mirror.form.min": "Sürekli çalışan",
|
||||
"mirror.form.max": "Yük altında en çok",
|
||||
"mirror.form.class": "Sınıf",
|
||||
@@ -209,7 +209,7 @@
|
||||
"mirror.err.replicas_whole": "replika sayıları tam sayı olmalı",
|
||||
"mirror.err.min_over_max": "sürekli çalışan (%d), yük altındaki en yüksek sayıdan (%d) fazla olamaz",
|
||||
"mirror.err.at_most": "en çok %d replika",
|
||||
"mirror.err.declared": "%s, server.yaml içinde tanımlı; orada değiştirin",
|
||||
"mirror.err.declared": "%s, cluster ayarlarında tanımlı; orada değiştirin (llmgw ctl config edit)",
|
||||
"mirror.err.no_model": "%q adında model yok",
|
||||
"mirror.err.nothing_fits": "daha fazlası sığmıyor",
|
||||
"mirror.err.no_room": "%s şu anda en çok %d replikayla sürekli çalışabilir (%d çalışıyor, %d tane daha için yer var): %s",
|
||||
|
||||
Binary file not shown.
@@ -261,7 +261,7 @@ func TestControlPlaneFeedsRouter(t *testing.T) {
|
||||
}()
|
||||
defer close(stop)
|
||||
|
||||
h.waitForEndpoints(t, 2, 15*time.Second)
|
||||
h.waitForEndpoints(t, 2, 30*time.Second)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
@@ -304,7 +304,7 @@ func TestRouterKeepsServingWhileNodeIsLost(t *testing.T) {
|
||||
}()
|
||||
defer close(stop)
|
||||
|
||||
h.waitForEndpoints(t, 2, 15*time.Second)
|
||||
h.waitForEndpoints(t, 2, 30*time.Second)
|
||||
|
||||
// Continuous traffic for the whole experiment.
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
|
||||
Reference in new issue
Block a user