Compare commits
4
Commits
v0.14.0
...
settings-e
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c52b73feec | ||
|
|
dc243f8840 | ||
|
|
659f20dc9a | ||
|
|
6bc6a67fb0 |
No files matched your search
+22
-2
@@ -807,14 +807,34 @@ Patroni configuration. The firewall's allowed set is the cluster's membership
|
||||
removed changes every machine's rules. Retires `firewall.Admit` and
|
||||
`firewall_networks` in server.yaml (imported into the store).
|
||||
|
||||
### E3. agent.yaml down to bootstrap (M)
|
||||
### E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch `settings-e`)
|
||||
agent.yaml keeps the control planes' addresses, its certificate and its id.
|
||||
Labels, fabrics, engines, heartbeat and supervisor settings move to the
|
||||
store per node and reach the agent over its stream; a change to the file is
|
||||
noticed and the agent restarts itself.
|
||||
Built as a section of the cluster's settings rather than a store of its own,
|
||||
so history, audit, `ctl config`, the console and the portal page cover it:
|
||||
`agents.default` and `agents.nodes.<id>` (config.AgentSettings; bootstrap
|
||||
keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge
|
||||
loaded as an agent would, so a bad fabric is refused on save). A live key:
|
||||
the control plane sends ControlMessage.AgentSettings on register and, a
|
||||
second apart per node, on change; the agent compares with what it started on,
|
||||
checks it loads beside its agent.yaml (else logs and keeps running), keeps it
|
||||
in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted).
|
||||
At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no
|
||||
longer load are set aside, never a reason not to start. agent.yaml is
|
||||
watched every 10 s and restarted on when it changes and loads. Left: file
|
||||
keys are not imported into the settings (they keep working under them);
|
||||
`node add` still writes a full agent.yaml; labels have two homes now
|
||||
(`ctl label`, which wins, and agents.*.labels).
|
||||
|
||||
### E4. Web settings page (S)
|
||||
### E4. Web settings page (S) — done 2026-10-08 (branch `settings-e`)
|
||||
The same document, edited on /admin/settings, for whoever has no terminal.
|
||||
Built: administrators only (staff roles get 404: webhook secrets), checked by
|
||||
the same function as PUT /api/v1/settings, made from a base version or
|
||||
refused on a newer one (the edit kept in the editor), history with view and
|
||||
roll back as a new version, audit with the size not the body, English and
|
||||
Turkish, a Settings link in the menu.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -24,6 +24,24 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **The cluster's settings on the portal** (BACKLOG E4): `/admin/settings`
|
||||
for administrators. It edits the document `llmgw ctl config edit` opens:
|
||||
checked as a control plane would load it, refused when made from a
|
||||
version since replaced, with the history to view and roll back. Changes
|
||||
are audited by size, never by contents.
|
||||
- **The nodes' agent settings in the cluster's settings** (BACKLOG E3):
|
||||
`agents.default` and `agents.nodes.<id>` hold everything agent.yaml can
|
||||
say but the agent's id, control planes, address and certificate. They are
|
||||
sent to the agents as they change; an agent starts again on new settings,
|
||||
or on an edited agent.yaml, with its engines serving throughout. Settings
|
||||
that would not load are refused on save, and set aside on a node rather
|
||||
than keep its agent from starting. Nodes need this release to follow
|
||||
them.
|
||||
|
||||
## v0.14.0 — 2026-10-08
|
||||
|
||||
Knowledge bases: a company's documents, put in by people from their own
|
||||
|
||||
@@ -3,8 +3,9 @@
|
||||
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
|
||||
NVLink domains and heterogeneous GPUs.
|
||||
|
||||
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
||||
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
||||
It serves LLMs running on your own GPU and CPU servers (vLLM, SGLang,
|
||||
llama.cpp, TensorRT-LLM) behind one OpenAI- and Anthropic-compatible
|
||||
endpoint, and answers from your own documents. Design document: [llmgw-plan.md](llmgw-plan.md).
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -37,6 +38,9 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.9.0 | — | Managing people and the tenancy tree | released |
|
||||
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
@@ -44,38 +48,60 @@ control plane places replicas — including ones larger than a single machine
|
||||
models load when someone asks for them and stop when nobody does, and keys,
|
||||
budgets and rate limits decide who may use any of it.
|
||||
|
||||
Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md),
|
||||
"Unreleased"):
|
||||
Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
|
||||
|
||||
- **Security hardening** — signed releases only, a token on the admin API,
|
||||
engines not run as root, TLS for the agent port, etcd and Patroni, a
|
||||
firewall for llmgw's internal ports, encrypted backups, bounded
|
||||
connections, sign-in throttling, a Content-Security-Policy.
|
||||
- **Guardrails** — policies that check requests and answers (PII, secrets,
|
||||
prompt injection, deny patterns, a judge model such as Llama Guard), in
|
||||
observe or enforce mode, attached to any part of the tenancy tree:
|
||||
- **Knowledge bases** (v0.14.0). People put a company's documents in from
|
||||
their own machines with `llmgw-kb`, a separate download for Windows,
|
||||
macOS and Linux that reads PDF, Office, OpenDocument, mail and many more
|
||||
formats and sends text and vectors, never the files. Collections are kept
|
||||
apart per tenant, down to row-level security in PostgreSQL. A chat
|
||||
request, or a part of the tree by default, is answered from them with
|
||||
its sources: [docs/knowledge.md](docs/knowledge.md).
|
||||
- **One command for the whole system** (v0.13.0): `llmgw stop`, `start` and
|
||||
`restart`. Qwen3Guard is the guardrails' judge, on a GPU and on CPUs
|
||||
together.
|
||||
- **Settings that apply when changed** (v0.12.0). The cluster's
|
||||
configuration lives in the store, the same on every control plane and in
|
||||
force within seconds (`llmgw ctl config`, the console's Settings). The
|
||||
firewall can be switched on and off for the whole cluster.
|
||||
- **Security hardening** (v0.11.2): signed releases only, a token on the
|
||||
admin API, engines not run as root, TLS for the agent port, etcd and
|
||||
Patroni, a firewall for llmgw's internal ports, encrypted backups,
|
||||
bounded connections, sign-in throttling, a Content-Security-Policy.
|
||||
- **Guardrails** (v0.11.2). Policies check requests and answers for PII,
|
||||
secrets, prompt injection, deny patterns, and with a judge model; they
|
||||
observe or enforce, and attach to any part of the tenancy tree:
|
||||
[docs/guardrails.md](docs/guardrails.md).
|
||||
- **Administration** — an audit log of every change (`ctl audit`), staff
|
||||
roles (viewer, operator, billing, admin), single sign-on through OpenID
|
||||
Connect and provisioning through SCIM 2.0, key lifetime limits and
|
||||
rotation with overlap (`keys ttl`, `keys rotate`), usage export as CSV
|
||||
(`ctl usage export`).
|
||||
- **Notifications** — webhooks, signed or Slack-shaped, for a model parked,
|
||||
a node lost, a budget spent and new weights published (`notify.webhooks`,
|
||||
`ctl notify test`), and Alertmanager receiver templates.
|
||||
- **Traffic and models** — request tags, canaries of new weights with
|
||||
automatic rollback, quality suites gating updates, a response cache,
|
||||
kept and replayed requests, batches, Anthropic's Messages API, more of
|
||||
OpenAI's (structured output, images, rerank, audio, moderations),
|
||||
scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, models
|
||||
imported from Ollama, routing by what each vLLM replica's cache holds,
|
||||
LoRA adapters loaded on demand, other llmgw clusters as providers, and
|
||||
placement that knows racks, PDUs and power caps.
|
||||
- **Operations** — `server restore` and off-machine backup copies, secrets
|
||||
in Vault, forgetting a person and keeping a project's requests in the
|
||||
cluster, offline installs from a bundle, a Helm chart, tracing from the
|
||||
control plane to the engines, the admin API in OpenAPI with a Terraform
|
||||
provider, and the portal in English and Turkish with its docs at /docs.
|
||||
- **Administration** (v0.11.2):
|
||||
- an audit log of every change (`ctl audit`);
|
||||
- staff roles (viewer, operator, billing, admin);
|
||||
- single sign-on through OpenID Connect and provisioning through SCIM
|
||||
2.0;
|
||||
- key lifetime limits, and rotation with overlap (`keys ttl`, `keys
|
||||
rotate`);
|
||||
- usage export as CSV (`ctl usage export`).
|
||||
- **Notifications** (v0.11.2): webhooks, signed or Slack-shaped, for a
|
||||
model parked, a node lost, a budget spent and new weights published, and
|
||||
Alertmanager receiver templates.
|
||||
- **Traffic and models** (v0.11.2):
|
||||
- request tags, canaries of new weights with automatic rollback, and
|
||||
quality suites gating updates;
|
||||
- a response cache, kept and replayed requests, and batches;
|
||||
- Anthropic's Messages API, and more of OpenAI's (structured output,
|
||||
images, rerank, audio, moderations);
|
||||
- scaling by the clock, and SLO-aware admission;
|
||||
- TensorRT-LLM, MIG, and models imported from Ollama;
|
||||
- routing by what each vLLM replica's cache holds, and LoRA adapters
|
||||
loaded on demand;
|
||||
- other llmgw clusters as providers, and placement that knows racks,
|
||||
PDUs and power caps.
|
||||
- **Operations** (v0.11.2):
|
||||
- `server restore`, off-machine backup copies, and secrets in Vault;
|
||||
- forgetting a person, and keeping a project's requests in the cluster;
|
||||
- offline installs from a bundle, and a Helm chart;
|
||||
- tracing from the control plane to the engines;
|
||||
- the admin API in OpenAPI, with a Terraform provider;
|
||||
- the portal in English and Turkish, with its docs at /docs.
|
||||
|
||||
Every one of these is in the console as well as on the command line:
|
||||
[docs/console.md](docs/console.md).
|
||||
@@ -124,6 +150,9 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
||||
official OpenAI and Anthropic SDKs, and llmgw's extensions
|
||||
- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
|
||||
policies, detectors, a judge model, and where each applies
|
||||
- [docs/knowledge.md](docs/knowledge.md) — knowledge bases: collections,
|
||||
who may read them, `llmgw-kb` on people's machines, and answers from
|
||||
documents
|
||||
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
|
||||
how many of each, and where the weights should live
|
||||
|
||||
@@ -152,7 +181,7 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
||||
## Building
|
||||
|
||||
```sh
|
||||
make build # bin/llmgw
|
||||
make build # bin/llmgw and bin/llmgw-kb
|
||||
make test # unit tests
|
||||
make race # with the race detector
|
||||
```
|
||||
@@ -169,6 +198,7 @@ llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
|
||||
sudo llmgw doctor # on a node: what its GPUs, NVLink and fabric need
|
||||
llmgw ctl audit --since 24h # who changed what
|
||||
llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model
|
||||
llmgw ctl kb list # knowledge-base collections, grants, defaults
|
||||
```
|
||||
|
||||
Bare `llmgw` opens the console, which does all of this from one screen.
|
||||
|
||||
+299
-226
File diff suppressed because it is too large.
Load diff
@@ -73,6 +73,7 @@ message ControlMessage {
|
||||
ControlPlanes control_planes = 11;
|
||||
Firewall firewall = 13;
|
||||
RestartAgent restart_agent = 14;
|
||||
AgentSettings agent_settings = 15;
|
||||
}
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -103,6 +104,17 @@ message Firewall {
|
||||
int32 engine_ports_to = 4;
|
||||
}
|
||||
|
||||
// AgentSettings is what the cluster's settings say of this node's agent
|
||||
// (agents.default, with agents.nodes.<id> over it), sent when it registers
|
||||
// and whenever they change. The agent keeps it beside agent.yaml, which
|
||||
// holds only what it needs to start; over that file, it is what the agent
|
||||
// runs on. A change from what it runs on has the agent start again on it,
|
||||
// its engines serving throughout.
|
||||
message AgentSettings {
|
||||
// yaml is agent.yaml's keys, as YAML; empty for none.
|
||||
string yaml = 1;
|
||||
}
|
||||
|
||||
// ControlPlanes is every control plane an agent may dial, sent by the leader
|
||||
// when an agent registers and whenever the list changes. The agent keeps it
|
||||
// beside its configuration, so a second control plane added after the node
|
||||
|
||||
+35
-2
@@ -258,8 +258,14 @@ llmgw ctl config history # who changed what, when, and why
|
||||
llmgw ctl config rollback 4 # back to version 4, as a new version
|
||||
```
|
||||
|
||||
or on the console's **Settings** screen. A change is checked before it is
|
||||
kept — one that would not load is refused — and recorded in the audit log.
|
||||
or on the console's **Settings** screen, or in a browser on the portal's
|
||||
**Settings** page (`/admin/settings`, for administrators only: the document
|
||||
can hold webhook secrets). The page edits the same document, lists its
|
||||
history, and puts an earlier version back. A change is checked before it is
|
||||
kept, so one that would not load is refused. A change made from a version
|
||||
someone else has replaced since is refused too, rather than overwriting
|
||||
theirs. Every change is recorded in the audit log, with the document's size
|
||||
but not its contents.
|
||||
Guardrails change live; anything else has every control plane start again on
|
||||
the new settings, the standbys first and the leader half a minute later, so
|
||||
one of them is always serving. Engines keep running through it.
|
||||
@@ -269,6 +275,33 @@ cluster keys — with a `baseline` guard policy, if the file had none — and
|
||||
from then on those keys in any server.yaml are not read (the server says so
|
||||
when it starts).
|
||||
|
||||
**The nodes' agents** are set the same way, under `agents`. `agents.default`
|
||||
is for every node, and `agents.nodes.<node-id>` is for one node, laid over
|
||||
the default:
|
||||
|
||||
```
|
||||
llmgw ctl config set agents.default.heartbeat_interval 5s
|
||||
llmgw ctl config set agents.nodes.cuda01.labels.gpu rtx6000
|
||||
llmgw ctl config edit # fabrics, engines, supervisor, discovery…
|
||||
```
|
||||
|
||||
Each node's `agent.yaml` then needs only what its agent must know before it
|
||||
can be told anything: `node_id`, `server` or `servers`, `hostname`,
|
||||
`mgmt_addr` and `tls`. Those keys are refused under `agents`.
|
||||
|
||||
- **Precedence.** Anything else in the file is still read, with the
|
||||
cluster's settings over it.
|
||||
- **Applying a change.** The control plane sends each node its settings, a
|
||||
second apart. An agent whose settings changed keeps them beside its
|
||||
`agent.yaml` (in `/var/lib/llmgw-agent/agent-settings.yaml`) and starts
|
||||
again on them, its engines serving throughout.
|
||||
- **Bad settings.** Settings that would not load are refused when they are
|
||||
saved. If they somehow do not load beside one node's own file, that agent
|
||||
keeps running on what it has and logs why.
|
||||
- **Editing `agent.yaml` itself.** The agent notices the change within ten
|
||||
seconds and starts again on it. If the edited file does not load, the
|
||||
agent reports it and keeps running on what it had.
|
||||
|
||||
**This machine's own keys** stay in `/etc/llmgw/server.yaml`: `listen`,
|
||||
`admin_listen`, `metrics_listen`, `agent_listen`, `agent_tls`, `auth.store`,
|
||||
`secrets`, `tracing`, `monitoring`, `mirror.api` and `mirror.tls`, the backup
|
||||
|
||||
@@ -26,6 +26,8 @@ rather than repeated.
|
||||
| `queue_capacity_per_band` | `int` | [Admission](#admission) |
|
||||
| `queue_capacity_total` | `int` | [Admission](#admission) |
|
||||
| `interval` | `time.Duration` | [Admission](#admission) |
|
||||
| `default` | `map[string]any` | [AgentSettings](#agentsettings) |
|
||||
| `nodes` | `map[string]map[string]any` | [AgentSettings](#agentsettings) |
|
||||
| `enabled` | `bool` | [Auth](#auth) |
|
||||
| `store` | `string` | [Auth](#auth) |
|
||||
| `cache_ttl` | `time.Duration` | [Auth](#auth) |
|
||||
@@ -294,6 +296,7 @@ rather than repeated.
|
||||
| `mail` | `mail.Config` | [Server](#server) |
|
||||
| `serving` | `ServingSettings` | [Server](#server) |
|
||||
| `firewall` | `FirewallSettings` | [Server](#server) |
|
||||
| `agents` | `AgentSettings` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
| `backup_copy_to` | `[]string` | [Server](#server) |
|
||||
@@ -469,6 +472,21 @@ Zero takes half of all the bands' capacities together.
|
||||
|
||||
`time.Duration`
|
||||
|
||||
## AgentSettings
|
||||
|
||||
### `default`
|
||||
|
||||
`map[string]any`
|
||||
|
||||
Default is agent.yaml's keys for every node.
|
||||
|
||||
### `nodes`
|
||||
|
||||
`map[string]map[string]any`
|
||||
|
||||
Nodes are agent.yaml's keys for one node each, by node id, over
|
||||
Default.
|
||||
|
||||
## Auth
|
||||
|
||||
### `enabled`
|
||||
@@ -2520,6 +2538,18 @@ Firewall turns llmgw's firewall on every machine on or off.
|
||||
|
||||
Takes a [FirewallSettings](#firewallsettings) block.
|
||||
|
||||
### `agents`
|
||||
|
||||
`AgentSettings`
|
||||
|
||||
Agents are the nodes' agent settings: agent.yaml's keys but those an
|
||||
agent needs to start, for every node (default) and for one
|
||||
(nodes.<id>). Sent to the agents as they change; an agent whose
|
||||
settings changed starts again on them, its engines serving
|
||||
throughout.
|
||||
|
||||
Takes a [AgentSettings](#agentsettings) block.
|
||||
|
||||
### `firewall_networks`
|
||||
|
||||
`[]string`
|
||||
|
||||
@@ -136,6 +136,10 @@ type Agent struct {
|
||||
// by mu.
|
||||
kv *kvevents.Mirror
|
||||
kvSubs map[string]*kvevents.Subscriber
|
||||
|
||||
// settingsCheck says whether the cluster's settings for this agent
|
||||
// would load beside its agent.yaml; see SetSettingsCheck.
|
||||
settingsCheck func([]byte) error
|
||||
}
|
||||
|
||||
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
|
||||
@@ -880,6 +884,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
|
||||
a.setControlPlanes(m.ControlPlanes)
|
||||
case *llmgwv1.ControlMessage_Firewall:
|
||||
go a.applyFirewall(m.Firewall)
|
||||
case *llmgwv1.ControlMessage_AgentSettings:
|
||||
go a.applySettings(m.AgentSettings)
|
||||
case *llmgwv1.ControlMessage_RestartAgent:
|
||||
go func() {
|
||||
a.log.Info("restarting, as the control plane asks; the engines keep serving",
|
||||
|
||||
@@ -103,6 +103,11 @@ type Config struct {
|
||||
Tracing obs.Tracing `koanf:"tracing"`
|
||||
|
||||
Version string
|
||||
|
||||
// Settings is the cluster's settings for this agent it was started on,
|
||||
// as the control plane sent them: what a newly sent version is
|
||||
// compared with.
|
||||
Settings string `koanf:"-"`
|
||||
}
|
||||
|
||||
// DefaultConfig returns sensible defaults.
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// The cluster's settings for this agent (BACKLOG E3): what agent.yaml can
|
||||
// say but the agent's id, control planes, address and certificate, kept by
|
||||
// the control planes and sent over the stream. Kept here beside agent.yaml,
|
||||
// which then holds only what the agent needs to start; at start the two are
|
||||
// read together, these over the file.
|
||||
|
||||
// SettingsFile keeps the settings the control plane last sent.
|
||||
var SettingsFile = "/var/lib/llmgw-agent/agent-settings.yaml"
|
||||
|
||||
// ReadSettings is what SettingsFile holds, nil when nothing.
|
||||
func ReadSettings() []byte {
|
||||
b, err := os.ReadFile(SettingsFile)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// SetSettingsCheck says how to tell whether settings would load beside this
|
||||
// agent's own file; settings that would not are not taken.
|
||||
func (a *Agent) SetSettingsCheck(f func([]byte) error) { a.settingsCheck = f }
|
||||
|
||||
// applySettings takes the settings the control plane sent: nothing when they
|
||||
// are what this agent runs on, and otherwise kept and started again on.
|
||||
func (a *Agent) applySettings(m *llmgwv1.AgentSettings) {
|
||||
next := m.GetYaml()
|
||||
if next == a.cfg.Settings {
|
||||
return
|
||||
}
|
||||
if a.settingsCheck != nil {
|
||||
if err := a.settingsCheck([]byte(next)); err != nil {
|
||||
// Kept running on what it has: settings that will not load
|
||||
// here must not take the node out of service.
|
||||
a.log.Error("the cluster's settings for this agent do not load beside its agent.yaml; running on what it has",
|
||||
"err", err)
|
||||
return
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(SettingsFile), 0o755); err != nil {
|
||||
a.log.Error("could not keep the cluster's settings for this agent", "err", err)
|
||||
return
|
||||
}
|
||||
tmp := SettingsFile + ".new"
|
||||
if err := os.WriteFile(tmp, []byte(next), 0o600); err != nil {
|
||||
a.log.Error("could not keep the cluster's settings for this agent", "err", err)
|
||||
return
|
||||
}
|
||||
if err := os.Rename(tmp, SettingsFile); err != nil {
|
||||
a.log.Error("could not keep the cluster's settings for this agent", "err", err)
|
||||
return
|
||||
}
|
||||
a.log.Info("the cluster's settings for this agent changed; starting again on them, the engines keep serving")
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
exitForUpgrade()
|
||||
}
|
||||
|
||||
// WatchFile starts the agent again when its configuration file changes and
|
||||
// still loads: agent.yaml is read only at start, and a change to it that
|
||||
// waited for somebody to restart the agent was a change nobody saw take
|
||||
// effect.
|
||||
func (a *Agent) WatchFile(ctx context.Context, path string, loads func() error) {
|
||||
if path == "" {
|
||||
return
|
||||
}
|
||||
sum := func() [32]byte {
|
||||
b, _ := os.ReadFile(path) //nolint:gosec // the agent's own configuration
|
||||
return sha256.Sum256(b)
|
||||
}
|
||||
was := sum()
|
||||
t := time.NewTicker(watchEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
now := sum()
|
||||
if now == was {
|
||||
continue
|
||||
}
|
||||
if err := loads(); err != nil {
|
||||
// Once per change: an editor saving halfway must not stop the
|
||||
// agent, and a broken file is reported, not restarted into.
|
||||
a.log.Error("agent.yaml changed and does not load; running on what it had", "path", path, "err", err)
|
||||
was = now
|
||||
continue
|
||||
}
|
||||
a.log.Info("agent.yaml changed; starting again on it, the engines keep serving", "path", path)
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
exitForUpgrade()
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// watchEvery is how often agent.yaml is looked at.
|
||||
var watchEvery = 10 * time.Second
|
||||
@@ -0,0 +1,88 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
func settingsAgent(t *testing.T, running string) (*Agent, chan struct{}) {
|
||||
t.Helper()
|
||||
oldFile, oldExit := SettingsFile, exitForUpgrade
|
||||
SettingsFile = filepath.Join(t.TempDir(), "agent-settings.yaml")
|
||||
exited := make(chan struct{}, 1)
|
||||
exitForUpgrade = func() { exited <- struct{}{} }
|
||||
t.Cleanup(func() { SettingsFile, exitForUpgrade = oldFile, oldExit })
|
||||
cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir(), Settings: running}
|
||||
cfg.Superv.WorkDir = t.TempDir()
|
||||
a, err := New(cfg, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return a, exited
|
||||
}
|
||||
|
||||
// exitedSoon is whether the agent exited within a second: it waits a
|
||||
// moment for its log line to leave first.
|
||||
func exitedSoon(ch chan struct{}) bool {
|
||||
select {
|
||||
case <-ch:
|
||||
return true
|
||||
case <-time.After(time.Second):
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAgentOnItsSettingsStaysAndOneOffThemStartsAgain(t *testing.T) {
|
||||
a, exited := settingsAgent(t, "heartbeat_interval: 5s\n")
|
||||
a.applySettings(&llmgwv1.AgentSettings{Yaml: "heartbeat_interval: 5s\n"})
|
||||
if exitedSoon(exited) || ReadSettings() != nil {
|
||||
t.Error("settings it already runs on restarted it")
|
||||
}
|
||||
a.applySettings(&llmgwv1.AgentSettings{Yaml: "heartbeat_interval: 2s\n"})
|
||||
if !exitedSoon(exited) || string(ReadSettings()) != "heartbeat_interval: 2s\n" {
|
||||
t.Errorf("new settings: kept %q", ReadSettings())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettingsThatWouldNotLoadAreNotTaken(t *testing.T) {
|
||||
a, exited := settingsAgent(t, "")
|
||||
a.SetSettingsCheck(func([]byte) error { return errors.New("fabric ib-a lists no HCAs") })
|
||||
a.applySettings(&llmgwv1.AgentSettings{Yaml: "fabrics: [{name: ib-a}]\n"})
|
||||
if exitedSoon(exited) || ReadSettings() != nil {
|
||||
t.Error("settings that do not load were taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAChangedAgentYamlThatLoadsStartsTheAgentAgain(t *testing.T) {
|
||||
a, exited := settingsAgent(t, "")
|
||||
old := watchEvery
|
||||
watchEvery = 10 * time.Millisecond
|
||||
t.Cleanup(func() { watchEvery = old })
|
||||
p := filepath.Join(t.TempDir(), "agent.yaml")
|
||||
_ = os.WriteFile(p, []byte("node_id: n1\n"), 0o600)
|
||||
var loads atomic.Bool
|
||||
check := func() error {
|
||||
if loads.Load() {
|
||||
return nil
|
||||
}
|
||||
return errors.New("broken")
|
||||
}
|
||||
go a.WatchFile(t.Context(), p, check)
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
// Saved broken: reported, not restarted into.
|
||||
_ = os.WriteFile(p, []byte("node_id: [\n"), 0o600)
|
||||
if exitedSoon(exited) {
|
||||
t.Fatal("a broken agent.yaml restarted the agent")
|
||||
}
|
||||
loads.Store(true)
|
||||
_ = os.WriteFile(p, []byte("node_id: n1\nlabels: {a: b}\n"), 0o600)
|
||||
if !exitedSoon(exited) {
|
||||
t.Error("a changed agent.yaml did not restart the agent")
|
||||
}
|
||||
}
|
||||
+22
-1
@@ -40,10 +40,23 @@ func newAgentCmd() *cobra.Command {
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
log := logger()
|
||||
|
||||
cfg, err := config.LoadAgent(configPath)
|
||||
// The cluster's settings for this agent over its file, as the
|
||||
// control plane last sent them. Settings that no longer load
|
||||
// beside the file are set aside rather than keeping the agent
|
||||
// from starting: the control plane sends them again, and says
|
||||
// why they are refused.
|
||||
settings := agent.ReadSettings()
|
||||
cfg, err := config.LoadAgentWith(configPath, settings)
|
||||
if err != nil && settings != nil {
|
||||
log.Warn("the cluster's settings kept for this agent do not load beside its agent.yaml; starting without them",
|
||||
"err", err)
|
||||
settings = nil
|
||||
cfg, err = config.LoadAgent(configPath)
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg.Settings = string(settings)
|
||||
if server != "" {
|
||||
cfg.Server = server
|
||||
}
|
||||
@@ -89,9 +102,17 @@ func newAgentCmd() *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
a.SetSettingsCheck(func(b []byte) error {
|
||||
_, err := config.LoadAgentWith(configPath, b)
|
||||
return err
|
||||
})
|
||||
|
||||
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
go a.WatchFile(ctx, configPath, func() error {
|
||||
_, err := config.LoadAgentWith(configPath, agent.ReadSettings())
|
||||
return err
|
||||
})
|
||||
|
||||
log.Info("agent starting",
|
||||
"node", cfg.NodeID, "server", cfg.Server,
|
||||
|
||||
@@ -241,6 +241,8 @@ var liveSettings = []string{
|
||||
"serving",
|
||||
// Sent to the nodes and this machine's firewall unit as they change.
|
||||
"firewall", "firewall_networks",
|
||||
// Sent to the nodes' agents as they change.
|
||||
"agents",
|
||||
// Read by server init, node add and backups, not by the server.
|
||||
"backup_recipients", "backup_copy_to",
|
||||
}
|
||||
|
||||
@@ -263,6 +263,15 @@ func newServerCmd() *cobra.Command {
|
||||
return &llmgwv1.Firewall{Enabled: c.Firewall.Enabled, Networks: c.FirewallNetworks,
|
||||
EnginePortsFrom: int32(from), EnginePortsTo: int32(to)} //nolint:gosec // ports
|
||||
})
|
||||
// What each node's agent runs on, as the settings say now.
|
||||
svc.SetAgentSettings(func(node string) string {
|
||||
b, err := config.Marshal(current.Load().Agents.For(node))
|
||||
if err != nil {
|
||||
log.Warn("could not write a node's agent settings", "node", node, "err", err)
|
||||
return ""
|
||||
}
|
||||
return string(b)
|
||||
})
|
||||
ctl.Reconciler().SetNotifier(notifier)
|
||||
admin = ctl
|
||||
agentPath, agentHandler = svc.Handler()
|
||||
@@ -382,6 +391,8 @@ func newServerCmd() *cobra.Command {
|
||||
// machine's own keys before it is kept.
|
||||
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
||||
Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
|
||||
// And the same document on the portal, for administrators.
|
||||
rt.SetSettingsCheck(func(body []byte) error { return checkClusterSettings(configPath, body) }, config.LocalKeys())
|
||||
}
|
||||
// Catalog changes need somewhere to persist, so this is wired
|
||||
// only when there is a store. Without one, the endpoints say so
|
||||
@@ -717,6 +728,10 @@ func newServerCmd() *cobra.Command {
|
||||
agentSvc.BroadcastFirewall()
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
|
||||
// A moment apart per node, so off this goroutine.
|
||||
go agentSvc.BroadcastAgentSettings()
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "guard") }) {
|
||||
if gate == nil {
|
||||
// Built at start, and only when there are policies.
|
||||
|
||||
@@ -18,7 +18,12 @@ import (
|
||||
//
|
||||
// An empty path loads the defaults, which with --server and --node-id on the
|
||||
// command line is enough for a smoke test on a node with no IB.
|
||||
func LoadAgent(path string) (agent.Config, error) {
|
||||
func LoadAgent(path string) (agent.Config, error) { return LoadAgentWith(path, nil) }
|
||||
|
||||
// LoadAgentWith is LoadAgent with the cluster's settings for this node
|
||||
// (agents.default and agents.nodes.<id>, as YAML) laid over the file: the
|
||||
// file is what the agent needs to start, the settings what it runs on.
|
||||
func LoadAgentWith(path string, settings []byte) (agent.Config, error) {
|
||||
cfg := agent.DefaultConfig()
|
||||
k := koanf.New(".")
|
||||
|
||||
@@ -30,6 +35,11 @@ func LoadAgent(path string) (agent.Config, error) {
|
||||
return cfg, fmt.Errorf("parsing %s: %w", path, err)
|
||||
}
|
||||
}
|
||||
if len(settings) > 0 {
|
||||
if err := k.Load(bytesProvider(settings), yaml.Parser()); err != nil {
|
||||
return cfg, fmt.Errorf("parsing the cluster's settings for this agent: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
err := k.Load(env.Provider(".", env.Opt{
|
||||
Prefix: "LLMGW_",
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// AgentSettings are the nodes' agent settings, kept with the cluster's
|
||||
// (BACKLOG E3): everything agent.yaml can say except what an agent needs to
|
||||
// start — its id, the control planes to dial, its address and its
|
||||
// certificate. Default applies to every node, and Nodes, by node id, over
|
||||
// it; both over the node's own agent.yaml.
|
||||
type AgentSettings struct {
|
||||
// Default is agent.yaml's keys for every node.
|
||||
Default map[string]any `koanf:"default"`
|
||||
// Nodes are agent.yaml's keys for one node each, by node id, over
|
||||
// Default.
|
||||
Nodes map[string]map[string]any `koanf:"nodes"`
|
||||
}
|
||||
|
||||
// agentBootstrapKeys are what an agent needs before it can be told anything:
|
||||
// they stay in agent.yaml.
|
||||
var agentBootstrapKeys = []string{"node_id", "server", "servers", "hostname", "mgmt_addr", "tls"}
|
||||
|
||||
// For is the settings for one node: Default with the node's own over it.
|
||||
// Empty when there are none.
|
||||
func (a AgentSettings) For(node string) map[string]any {
|
||||
out := clone(stringKeys(a.Default))
|
||||
if n, ok := a.Nodes[node]; ok {
|
||||
merge(out, stringKeys(n))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Check refuses settings that set an agent's bootstrap keys, or that would
|
||||
// not load as an agent's configuration for some node.
|
||||
func (a AgentSettings) Check() error {
|
||||
check := func(where string, m map[string]any) error {
|
||||
for _, k := range agentBootstrapKeys {
|
||||
if _, ok := m[k]; ok {
|
||||
return fmt.Errorf("%s.%s stays in each node's agent.yaml: an agent needs it before it can be told anything", where, k)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err := check("agents.default", stringKeys(a.Default)); err != nil {
|
||||
return err
|
||||
}
|
||||
nodes := make([]string, 0, len(a.Nodes))
|
||||
for n := range a.Nodes {
|
||||
nodes = append(nodes, n)
|
||||
}
|
||||
sort.Strings(nodes)
|
||||
for _, n := range nodes {
|
||||
if err := check("agents.nodes."+n, stringKeys(a.Nodes[n])); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Each node's settings as its agent would load them, over nothing: what
|
||||
// its own file adds cannot be known here, and what is wrong in the
|
||||
// settings alone is wrong on every node.
|
||||
for _, n := range append([]string{""}, nodes...) {
|
||||
b, err := Marshal(a.For(n))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := LoadAgentWith("", b); err != nil {
|
||||
where := "agents.default"
|
||||
if n != "" {
|
||||
where = "agents.nodes." + n
|
||||
}
|
||||
return fmt.Errorf("%s: %w", where, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// AgentBootstrapKeys are the keys agent.yaml keeps, for saying so.
|
||||
func AgentBootstrapKeys() []string { return slices.Clone(agentBootstrapKeys) }
|
||||
|
||||
// stringKeys turns YAML's map[any]any, wherever it hides, into the
|
||||
// map[string]any the rest of this package merges.
|
||||
func stringKeys(m map[string]any) map[string]any {
|
||||
out := make(map[string]any, len(m))
|
||||
for k, v := range m {
|
||||
out[k] = normalize(v)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func normalize(v any) any {
|
||||
switch t := v.(type) {
|
||||
case map[string]any:
|
||||
return stringKeys(t)
|
||||
case map[any]any:
|
||||
m := make(map[string]any, len(t))
|
||||
for k, vv := range t {
|
||||
m[fmt.Sprint(k)] = normalize(vv)
|
||||
}
|
||||
return m
|
||||
case []any:
|
||||
out := make([]any, len(t))
|
||||
for i, vv := range t {
|
||||
out[i] = normalize(vv)
|
||||
}
|
||||
return out
|
||||
}
|
||||
return v
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestANodesAgentSettingsAreTheDefaultWithItsOwnOver(t *testing.T) {
|
||||
a := AgentSettings{
|
||||
Default: map[string]any{"heartbeat_interval": "5s", "supervisor": map[string]any{"work_dir": "/srv/w", "x": 1}},
|
||||
Nodes: map[string]map[string]any{
|
||||
"cuda01": {"supervisor": map[string]any{"work_dir": "/fast/w"}, "labels": map[string]any{"gpu": "rtx6000"}},
|
||||
},
|
||||
}
|
||||
got := a.For("cuda01")
|
||||
sup := got["supervisor"].(map[string]any)
|
||||
if sup["work_dir"] != "/fast/w" || sup["x"] != 1 || got["heartbeat_interval"] != "5s" || got["labels"] == nil {
|
||||
t.Errorf("cuda01: %v", got)
|
||||
}
|
||||
if got := a.For("node01"); got["labels"] != nil || got["supervisor"].(map[string]any)["work_dir"] != "/srv/w" {
|
||||
t.Errorf("node01: %v", got)
|
||||
}
|
||||
if got := (AgentSettings{}).For("x"); len(got) != 0 {
|
||||
t.Errorf("none: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentSettingsKeepBootstrapKeysInTheFileAndMustLoad(t *testing.T) {
|
||||
for _, a := range []AgentSettings{
|
||||
{Default: map[string]any{"servers": []any{"x:7443"}}},
|
||||
{Nodes: map[string]map[string]any{"n1": {"node_id": "other"}}},
|
||||
{Nodes: map[string]map[string]any{"n1": {"tls": map[string]any{"ca": "/x"}}}},
|
||||
} {
|
||||
if err := a.Check(); err == nil || !strings.Contains(err.Error(), "stays in each node's agent.yaml") {
|
||||
t.Errorf("%+v: %v", a, err)
|
||||
}
|
||||
}
|
||||
// A fabric with no adapters would not load on the node it names.
|
||||
bad := AgentSettings{Nodes: map[string]map[string]any{"n1": {"fabrics": []any{map[string]any{"name": "ib-a"}}}}}
|
||||
if err := bad.Check(); err == nil || !strings.Contains(err.Error(), "agents.nodes.n1") || !strings.Contains(err.Error(), "no HCAs") {
|
||||
t.Errorf("a fabric without HCAs: %v", err)
|
||||
}
|
||||
if err := (AgentSettings{Default: map[string]any{"heartbeat_interval": "3s"}}).Check(); err != nil {
|
||||
t.Errorf("a plain setting: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheClusterSettingsCarryAgentsAndRefuseBadOnes(t *testing.T) {
|
||||
good, err := Parse([]byte("agents:\n default:\n heartbeat_interval: 3s\n nodes:\n cuda01:\n labels: {gpu: rtx6000}\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg, err := FromMap(good)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.Agents.For("cuda01")["labels"] == nil {
|
||||
t.Errorf("agents not read: %+v", cfg.Agents)
|
||||
}
|
||||
bad, _ := Parse([]byte("agents:\n default:\n mgmt_addr: 10.0.0.1\n"))
|
||||
if _, err := FromMap(bad); err == nil {
|
||||
t.Error("a bootstrap key in the cluster's settings was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSettingsAreLaidOverTheAgentsFile(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "agent.yaml")
|
||||
_ = os.WriteFile(p, []byte("node_id: n1\nserver: cp:7443\nheartbeat_interval: 9s\nlabels: {rack: r1}\n"), 0o600)
|
||||
cfg, err := LoadAgentWith(p, []byte("heartbeat_interval: 2s\nlabels: {gpu: a100}\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.NodeID != "n1" || cfg.HeartbeatInterval != 2*time.Second || cfg.Labels["rack"] != "r1" || cfg.Labels["gpu"] != "a100" {
|
||||
t.Errorf("loaded %+v", cfg)
|
||||
}
|
||||
}
|
||||
@@ -158,6 +158,13 @@ type Server struct {
|
||||
// Firewall turns llmgw's firewall on every machine on or off.
|
||||
Firewall FirewallSettings `koanf:"firewall"`
|
||||
|
||||
// Agents are the nodes' agent settings: agent.yaml's keys but those an
|
||||
// agent needs to start, for every node (default) and for one
|
||||
// (nodes.<id>). Sent to the agents as they change; an agent whose
|
||||
// settings changed starts again on them, its engines serving
|
||||
// throughout.
|
||||
Agents AgentSettings `koanf:"agents"`
|
||||
|
||||
// FirewallNetworks are networks, besides those each machine reaches the
|
||||
// cluster on, that llmgw's internal ports answer: a monitoring host on
|
||||
// another subnet, nodes behind a router. Written into every machine's
|
||||
@@ -887,6 +894,9 @@ func load(b []byte) (Server, error) {
|
||||
// Validate rejects configurations that would otherwise fail confusingly at
|
||||
// request time.
|
||||
func (s Server) Validate() error {
|
||||
if err := s.Agents.Check(); err != nil {
|
||||
return err
|
||||
}
|
||||
for i, o := range s.EngineOverrides {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// SetAgentSettings gives the service each node's agent settings from the
|
||||
// cluster's (BACKLOG E3), as YAML, read at each use so a change is what is
|
||||
// sent next.
|
||||
func (s *Service) SetAgentSettings(f func(node string) string) {
|
||||
s.mu.Lock()
|
||||
s.agentSettings = f
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Service) agentSettingsMsg(node string) *llmgwv1.ControlMessage {
|
||||
s.mu.RLock()
|
||||
f := s.agentSettings
|
||||
s.mu.RUnlock()
|
||||
if f == nil {
|
||||
return nil
|
||||
}
|
||||
return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_AgentSettings{
|
||||
AgentSettings: &llmgwv1.AgentSettings{Yaml: f(node)}}}
|
||||
}
|
||||
|
||||
// agentSettingsGap is the pause between telling one node and the next: an
|
||||
// agent whose settings changed starts again, and a whole cluster's agents
|
||||
// doing so in the same second would all re-register at once.
|
||||
var agentSettingsGap = time.Second
|
||||
|
||||
// BroadcastAgentSettings tells every connected agent what the cluster's
|
||||
// settings now say it runs on, a moment apart. A node not connected is told
|
||||
// when it registers.
|
||||
func (s *Service) BroadcastAgentSettings() {
|
||||
for _, node := range s.ConnectedNodes() {
|
||||
msg := s.agentSettingsMsg(node)
|
||||
if msg == nil {
|
||||
return
|
||||
}
|
||||
if err := s.send(node, msg); err != nil {
|
||||
s.log.Warn("could not tell a node its agent settings; it is told when it next registers",
|
||||
"node", node, "err", err)
|
||||
continue
|
||||
}
|
||||
time.Sleep(agentSettingsGap)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestEachNodeIsToldItsOwnAgentSettings(t *testing.T) {
|
||||
s := New(inventory.NewStore(), nil, false, slog.New(slog.DiscardHandler))
|
||||
if s.agentSettingsMsg("n1") != nil {
|
||||
t.Error("told settings with nothing to read them from")
|
||||
}
|
||||
s.SetAgentSettings(func(node string) string {
|
||||
if node == "cuda01" {
|
||||
return "labels:\n gpu: rtx6000\n"
|
||||
}
|
||||
return ""
|
||||
})
|
||||
if got := s.agentSettingsMsg("cuda01").GetAgentSettings().GetYaml(); got != "labels:\n gpu: rtx6000\n" {
|
||||
t.Errorf("cuda01 told %q", got)
|
||||
}
|
||||
// None is said too, so an agent whose settings were removed drops them.
|
||||
if m := s.agentSettingsMsg("node01"); m == nil || m.GetAgentSettings().GetYaml() != "" {
|
||||
t.Errorf("node01 told %v", m)
|
||||
}
|
||||
}
|
||||
@@ -79,6 +79,9 @@ type Service struct {
|
||||
controlPlanes func(context.Context) ([]string, error)
|
||||
// firewall is what nodes are told of llmgw's firewall; see SetFirewall.
|
||||
firewall func() *llmgwv1.Firewall
|
||||
// agentSettings is what each node's agent is told to run on; see
|
||||
// SetAgentSettings.
|
||||
agentSettings func(node string) string
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -296,6 +299,13 @@ func (s *Service) Connect(
|
||||
return err
|
||||
}
|
||||
}
|
||||
// What the cluster's settings say this agent runs on. An agent already
|
||||
// on them ignores it; one that is not starts again on them.
|
||||
if msg := s.agentSettingsMsg(nodeID); msg != nil {
|
||||
if err := stream.Send(msg); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Recover replica state from what this node is already running, before any
|
||||
// reconciliation pass can conclude the cluster is empty and place
|
||||
|
||||
@@ -19,6 +19,10 @@ var Reference = []Block{
|
||||
{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued across bands. Reached, a\nrequest is not refused while a lower band has any queued: the newest of\nthe lowest band is turned away instead, with a 503 and Retry-After.\nZero takes half of all the bands' capacities together.", Nested: ""},
|
||||
{Key: "interval", Type: "time.Duration", Doc: "", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "AgentSettings", Title: "AgentSettings", Doc: "", Fields: []Field{
|
||||
{Key: "default", Type: "map[string]any", Doc: "Default is agent.yaml's keys for every node.", Nested: ""},
|
||||
{Key: "nodes", Type: "map[string]map[string]any", Doc: "Nodes are agent.yaml's keys for one node each, by node id, over\nDefault.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Auth", Title: "Auth", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "store", Type: "string", Doc: "Store is the SQLite file holding keys, tenancy and usage.", Nested: ""},
|
||||
@@ -372,6 +376,7 @@ var Reference = []Block{
|
||||
{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
|
||||
{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
|
||||
{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
|
||||
{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
|
||||
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
|
||||
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
|
||||
{Key: "backup_copy_to", Type: "[]string", Doc: "BackupCopyTo is where each archive is copied once written, so it\nsurvives this machine: a directory (a mounted share, another disk) or\nssh://user@host[:port]/path, reached with BackupCopyKey and checked\nagainst /etc/llmgw/backup_known_hosts, which learns a host the first\ntime. The same number of archives is kept there as here. Refused\nwithout backup_recipients: an archive leaves the machine encrypted.", Nested: ""},
|
||||
|
||||
@@ -32,6 +32,7 @@ var Sources = []string{
|
||||
"internal/config/oidc.go",
|
||||
"internal/config/firewall.go",
|
||||
"internal/config/serving.go",
|
||||
"internal/config/agents.go",
|
||||
"internal/mail/mail.go",
|
||||
"internal/notify/notify.go",
|
||||
"internal/secrets/secrets.go",
|
||||
|
||||
@@ -247,6 +247,9 @@ func (r *Router) Handler() http.Handler {
|
||||
if r.store != nil && r.guardLibrary != nil {
|
||||
pages.RegisterAdminGuard(mux, r.store, r.guardLibrary)
|
||||
}
|
||||
if r.store != nil && r.settingsCheck != nil {
|
||||
pages.RegisterAdminSettings(mux, r.store, r.settingsCheck, r.settingsLocal)
|
||||
}
|
||||
pages.Register(mux)
|
||||
// The docs portal, with the menu of whoever is signed in, and only to
|
||||
// them when the site asked for that.
|
||||
|
||||
@@ -177,6 +177,10 @@ type Router struct {
|
||||
// collections for kbDefaultTTL.
|
||||
kbDefMu sync.Mutex
|
||||
kbDefCache map[string]kbDefaultEntry
|
||||
|
||||
// settingsCheck and settingsLocal mount the portal's settings page.
|
||||
settingsCheck func([]byte) error
|
||||
settingsLocal []string
|
||||
}
|
||||
|
||||
// SetClusterView gives the router the data the /admin page renders.
|
||||
@@ -200,6 +204,13 @@ func (r *Router) SetGuard(g guard.Guard) { r.guard = g }
|
||||
// SetGuardLibrary is the policies the guard was built from, for the pages.
|
||||
func (r *Router) SetGuardLibrary(c func() guard.Config) { r.guardLibrary = c }
|
||||
|
||||
// SetSettingsCheck enables the portal's settings page: check says whether a
|
||||
// document would load as the cluster's settings beside this machine's own
|
||||
// server.yaml, and local are the keys that file keeps.
|
||||
func (r *Router) SetSettingsCheck(check func([]byte) error, local []string) {
|
||||
r.settingsCheck, r.settingsLocal = check, local
|
||||
}
|
||||
|
||||
// SetStopped refuses, or again serves, every request for a model.
|
||||
func (r *Router) SetStopped(stopped bool) { r.stopped.Store(stopped) }
|
||||
|
||||
|
||||
@@ -90,7 +90,9 @@ type AccountPages struct {
|
||||
audit audit.Sink
|
||||
// hasGuard is set when /admin/guard is registered, and guardLibrary
|
||||
// with it: the policies in force, whose judges the mirror page serves.
|
||||
hasGuard bool
|
||||
hasGuard bool
|
||||
// hasSettings is set when /admin/settings is registered.
|
||||
hasSettings bool
|
||||
guardLibrary func() guard.Config
|
||||
// guardFor names the guardrail policies in force for someone, and
|
||||
// whether they keep nothing; nil when there are none.
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// The cluster's settings, edited in the browser (BACKLOG E4): the document
|
||||
// `llmgw ctl config edit` opens, for whoever has no terminal on a control
|
||||
// plane. The same rules hold here as there — a version is checked as a
|
||||
// control plane would load it before it is kept, it is made from the newest
|
||||
// version or refused, and every change is in force on every control plane
|
||||
// within seconds and in the audit log.
|
||||
|
||||
// SettingsAdmin is where the cluster's settings are kept.
|
||||
type SettingsAdmin interface {
|
||||
ClusterSettings(ctx context.Context) (store.SettingsVersion, error)
|
||||
ClusterSettingsAt(ctx context.Context, version int64) (store.SettingsVersion, error)
|
||||
ClusterSettingsHistory(ctx context.Context, limit int) ([]store.SettingsVersion, error)
|
||||
PutClusterSettings(ctx context.Context, base int64, body, by, note string) (int64, error)
|
||||
}
|
||||
|
||||
type settingsPageData struct {
|
||||
Nav Nav
|
||||
Version string
|
||||
CSRF string
|
||||
// Current is the version in force; Shown the one in the editor, which
|
||||
// is an earlier one when ?version= asked for it.
|
||||
Current, Shown store.SettingsVersion
|
||||
Earlier bool
|
||||
// Body is what the editor holds: the version shown, or what was sent
|
||||
// and refused, so a refusal does not lose the edit.
|
||||
Body string
|
||||
Note string
|
||||
History []settingsRow
|
||||
LocalKeys []string
|
||||
Notice string
|
||||
Error string
|
||||
}
|
||||
|
||||
type settingsRow struct {
|
||||
Version int64
|
||||
At, By string
|
||||
Note string
|
||||
InForce bool
|
||||
Selected bool
|
||||
}
|
||||
|
||||
// RegisterAdminSettings mounts /admin/settings, for administrators only:
|
||||
// the settings hold webhook addresses and the like, so even reading them is
|
||||
// not every staff role's.
|
||||
func (p *AccountPages) RegisterAdminSettings(mux *http.ServeMux, s SettingsAdmin, check func([]byte) error, localKeys []string) {
|
||||
p.hasSettings = true
|
||||
page := func(w http.ResponseWriter, r *http.Request, admin store.User, d settingsPageData) {
|
||||
ctx, l := r.Context(), langOf(r)
|
||||
d.Nav, d.Version, d.CSRF, d.LocalKeys = p.navFor(r, admin, "settings"), p.version, csrfFor(r), localKeys
|
||||
cur, err := s.ClusterSettings(ctx)
|
||||
if err != nil {
|
||||
d.Error = l.Err(err)
|
||||
}
|
||||
d.Current = cur
|
||||
if d.Shown.Version == 0 {
|
||||
d.Shown = cur
|
||||
}
|
||||
d.Earlier = d.Shown.Version != cur.Version
|
||||
if d.Body == "" {
|
||||
d.Body = d.Shown.Body
|
||||
}
|
||||
if hist, err := s.ClusterSettingsHistory(ctx, 30); err == nil {
|
||||
for _, v := range hist {
|
||||
d.History = append(d.History, settingsRow{Version: v.Version, At: l.Date(v.At.UTC()) + " " + v.At.UTC().Format("15:04"),
|
||||
By: v.By, Note: v.Note, InForce: v.Version == cur.Version, Selected: v.Version == d.Shown.Version})
|
||||
}
|
||||
}
|
||||
noStore(w)
|
||||
render(w, r, "admin_settings", d)
|
||||
}
|
||||
adminsOnly := func(next func(w http.ResponseWriter, r *http.Request, admin store.User)) http.HandlerFunc {
|
||||
return p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
if !admin.IsAdmin {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
if r.Method == http.MethodPost {
|
||||
sess, _, _ := p.session(r)
|
||||
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
}
|
||||
next(w, r, admin)
|
||||
})
|
||||
}
|
||||
|
||||
mux.HandleFunc("GET /admin/settings", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
var d settingsPageData
|
||||
if q := r.URL.Query().Get("version"); q != "" {
|
||||
n, err := strconv.ParseInt(q, 10, 64)
|
||||
if v, verr := s.ClusterSettingsAt(r.Context(), n); err == nil && verr == nil && v.Version != 0 {
|
||||
d.Shown = v
|
||||
} else {
|
||||
d.Error = langOf(r).T("settings.err.no_version")
|
||||
}
|
||||
}
|
||||
page(w, r, admin, d)
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/settings", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
l := langOf(r)
|
||||
body := strings.ReplaceAll(r.FormValue("body"), "\r\n", "\n")
|
||||
note := strings.TrimSpace(r.FormValue("note"))
|
||||
base, _ := strconv.ParseInt(r.FormValue("base"), 10, 64)
|
||||
// The audit line records the form; the document can hold webhook
|
||||
// secrets, so it records the document's size, not the document.
|
||||
r.PostForm.Set("body", "("+strconv.Itoa(len(body))+" bytes)")
|
||||
refuse := func(msg string) {
|
||||
noteAudit(r, "cluster settings", "", msg)
|
||||
page(w, r, admin, settingsPageData{Body: body, Note: note, Error: msg})
|
||||
}
|
||||
if cur, err := s.ClusterSettings(r.Context()); err == nil && cur.Body == body {
|
||||
refuse(l.T("settings.err.unchanged"))
|
||||
return
|
||||
}
|
||||
if err := check([]byte(body)); err != nil {
|
||||
refuse(l.T("settings.err.invalid") + " " + err.Error())
|
||||
return
|
||||
}
|
||||
n, err := s.PutClusterSettings(r.Context(), base, body, admin.Email, note)
|
||||
switch {
|
||||
case errors.Is(err, store.ErrSettingsChanged):
|
||||
refuse(l.T("settings.err.changed"))
|
||||
return
|
||||
case err != nil:
|
||||
refuse(l.Err(err))
|
||||
return
|
||||
}
|
||||
noteAudit(r, "cluster settings", "version "+strconv.FormatInt(n, 10), "")
|
||||
page(w, r, admin, settingsPageData{Notice: l.T("settings.saved", n)})
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/settings/rollback", adminsOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
l := langOf(r)
|
||||
n, _ := strconv.ParseInt(r.FormValue("version"), 10, 64)
|
||||
old, err := s.ClusterSettingsAt(r.Context(), n)
|
||||
if err != nil || old.Version == 0 {
|
||||
page(w, r, admin, settingsPageData{Error: l.T("settings.err.no_version")})
|
||||
return
|
||||
}
|
||||
cur, err := s.ClusterSettings(r.Context())
|
||||
if err != nil {
|
||||
page(w, r, admin, settingsPageData{Error: l.Err(err)})
|
||||
return
|
||||
}
|
||||
if err := check([]byte(old.Body)); err != nil {
|
||||
// A version kept before a change to server.yaml may no longer
|
||||
// load beside it; refused here rather than breaking the cluster.
|
||||
noteAudit(r, "cluster settings", "", err.Error())
|
||||
page(w, r, admin, settingsPageData{Error: l.T("settings.err.invalid") + " " + err.Error()})
|
||||
return
|
||||
}
|
||||
v, err := s.PutClusterSettings(r.Context(), cur.Version, old.Body, admin.Email, "back to version "+strconv.FormatInt(n, 10))
|
||||
if err != nil {
|
||||
page(w, r, admin, settingsPageData{Error: l.Err(err)})
|
||||
return
|
||||
}
|
||||
noteAudit(r, "cluster settings", "version "+strconv.FormatInt(v, 10)+" from "+strconv.FormatInt(n, 10), "")
|
||||
page(w, r, admin, settingsPageData{Notice: l.T("settings.rolled_back", n, v)})
|
||||
}))
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeSettings struct{ versions []store.SettingsVersion }
|
||||
|
||||
func (f *fakeSettings) ClusterSettings(context.Context) (store.SettingsVersion, error) {
|
||||
if len(f.versions) == 0 {
|
||||
return store.SettingsVersion{}, nil
|
||||
}
|
||||
return f.versions[len(f.versions)-1], nil
|
||||
}
|
||||
|
||||
func (f *fakeSettings) ClusterSettingsAt(_ context.Context, v int64) (store.SettingsVersion, error) {
|
||||
for _, s := range f.versions {
|
||||
if s.Version == v {
|
||||
return s, nil
|
||||
}
|
||||
}
|
||||
return store.SettingsVersion{}, nil
|
||||
}
|
||||
|
||||
func (f *fakeSettings) ClusterSettingsHistory(context.Context, int) ([]store.SettingsVersion, error) {
|
||||
out := make([]store.SettingsVersion, len(f.versions))
|
||||
for i, v := range f.versions {
|
||||
out[len(f.versions)-1-i] = v
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (f *fakeSettings) PutClusterSettings(_ context.Context, base int64, body, by, note string) (int64, error) {
|
||||
cur, _ := f.ClusterSettings(context.Background())
|
||||
if base != cur.Version {
|
||||
return 0, store.ErrSettingsChanged
|
||||
}
|
||||
n := cur.Version + 1
|
||||
f.versions = append(f.versions, store.SettingsVersion{Version: n, Body: body, By: by, Note: note, At: time.Now()})
|
||||
return n, nil
|
||||
}
|
||||
|
||||
type auditSink struct{ events []store.AuditEvent }
|
||||
|
||||
func (a *auditSink) WriteAudit(_ context.Context, e store.AuditEvent) error {
|
||||
a.events = append(a.events, e)
|
||||
return nil
|
||||
}
|
||||
|
||||
func clusterSettingsPage(t *testing.T, user store.User) (*http.ServeMux, *fakeSettings, *auditSink) {
|
||||
t.Helper()
|
||||
s := &fakeSettings{versions: []store.SettingsVersion{{Version: 1, Body: "guard:\n enabled: true\n", By: "init", At: time.Now()}}}
|
||||
p := NewAccountPages(fakeAccounts{user: user}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
sink := &auditSink{}
|
||||
p.SetAudit(sink)
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminSettings(mux, s, func(b []byte) error {
|
||||
if strings.Contains(string(b), "bogus") {
|
||||
return errors.New("unknown key bogus")
|
||||
}
|
||||
return nil
|
||||
}, []string{"listen", "store"})
|
||||
return mux, s, sink
|
||||
}
|
||||
|
||||
func TestTheSettingsPageIsForAdministratorsOnly(t *testing.T) {
|
||||
for _, u := range []store.User{
|
||||
{ID: "u1", Email: "v@example.com", Role: store.RoleViewer},
|
||||
{ID: "u2", Email: "o@example.com", Role: store.RoleOperator},
|
||||
{ID: "u3", Email: "b@example.com", Role: store.RoleBilling},
|
||||
} {
|
||||
mux, _, _ := clusterSettingsPage(t, u)
|
||||
if w := mirrorGet(t, mux, "/admin/settings"); w.Code != http.StatusNotFound {
|
||||
t.Errorf("%s saw the settings: %d", u.Role, w.Code)
|
||||
}
|
||||
}
|
||||
mux, _, _ := clusterSettingsPage(t, store.User{ID: "a", Email: "a@example.com", IsAdmin: true})
|
||||
body := mirrorGet(t, mux, "/admin/settings").Body.String()
|
||||
for _, want := range []string{"guard:\n enabled: true", "Version 1, in force", "<code>listen</code>", `href="/admin/settings" aria-current="page"`} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("no %q in:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettingsAreCheckedAppliedAndRolledBack(t *testing.T) {
|
||||
mux, s, sink := clusterSettingsPage(t, store.User{ID: "a", Email: "a@example.com", IsAdmin: true})
|
||||
post := func(form url.Values) string {
|
||||
form.Set("csrf", "tok")
|
||||
return mirrorPost(t, mux, "/admin/settings", form).Body.String()
|
||||
}
|
||||
// Refused: does not load. The edit stays in the editor; nothing kept.
|
||||
body := post(url.Values{"base": {"1"}, "body": {"bogus: 1\n"}})
|
||||
if !strings.Contains(body, "unknown key bogus") || !strings.Contains(body, "bogus: 1") || len(s.versions) != 1 {
|
||||
t.Errorf("an invalid document: %d versions\n%s", len(s.versions), body)
|
||||
}
|
||||
// Applied, with its note and who.
|
||||
body = post(url.Values{"base": {"1"}, "body": {"guard:\r\n enabled: false\r\n"}, "note": {"maintenance"}})
|
||||
if !strings.Contains(body, "Version 2 is in force") || len(s.versions) != 2 ||
|
||||
s.versions[1].Body != "guard:\n enabled: false\n" || s.versions[1].By != "a@example.com" || s.versions[1].Note != "maintenance" {
|
||||
t.Errorf("applying: %+v\n%s", s.versions, body)
|
||||
}
|
||||
// Made from a version no longer the newest: refused, not overwritten.
|
||||
body = post(url.Values{"base": {"1"}, "body": {"guard:\n enabled: true\n x: 1\n"}})
|
||||
if !strings.Contains(body, "someone changed the settings") || len(s.versions) != 2 {
|
||||
t.Errorf("a stale edit: %d versions\n%s", len(s.versions), body)
|
||||
}
|
||||
// An earlier version is shown, and put back as a new one.
|
||||
if body := mirrorGet(t, mux, "/admin/settings?version=1").Body.String(); !strings.Contains(body, "Put version 1 back in force") {
|
||||
t.Errorf("viewing version 1:\n%s", body)
|
||||
}
|
||||
w := mirrorPost(t, mux, "/admin/settings/rollback", url.Values{"csrf": {"tok"}, "version": {"1"}})
|
||||
if len(s.versions) != 3 || s.versions[2].Body != s.versions[0].Body || !strings.Contains(w.Body.String(), "as version 3") {
|
||||
t.Errorf("rollback: %+v\n%s", s.versions, w.Body.String())
|
||||
}
|
||||
// Without the CSRF token, nothing.
|
||||
if w := mirrorPost(t, mux, "/admin/settings", url.Values{"base": {"3"}, "body": {"a: 1\n"}}); w.Code != http.StatusBadRequest || len(s.versions) != 3 {
|
||||
t.Errorf("no token: %d", w.Code)
|
||||
}
|
||||
// Audited, without the document itself: it can hold secrets.
|
||||
var saw bool
|
||||
for _, e := range sink.events {
|
||||
if strings.Contains(e.Detail, "enabled") {
|
||||
t.Errorf("the document is in the audit log: %+v", e)
|
||||
}
|
||||
if e.Action == "POST /admin/settings" && strings.Contains(e.Detail, "version 2") {
|
||||
saw = true
|
||||
}
|
||||
}
|
||||
if !saw {
|
||||
t.Errorf("no audit line for version 2: %+v", sink.events)
|
||||
}
|
||||
}
|
||||
@@ -10,5 +10,6 @@
|
||||
"nav.chat": "Chat",
|
||||
"nav.account": "Account",
|
||||
"nav.sign_out": "Sign out",
|
||||
"nav.docs": "Docs"
|
||||
"nav.docs": "Docs",
|
||||
"nav.settings": "Settings"
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"settings.title": "Settings",
|
||||
"settings.sub": "The cluster's settings: everything that is not one machine's own, the same on every control plane, in force within seconds of a change. <code>llmgw ctl config edit</code> edits the same document.",
|
||||
"settings.current": "Version %d, in force",
|
||||
"settings.earlier": "Version %d",
|
||||
"settings.earlier_sub": "An earlier version; version %d is in force. Rolling back keeps this one as a new version, so nothing is lost.",
|
||||
"settings.rollback": "Put version %d back in force",
|
||||
"settings.back_to_current": "back to the version in force",
|
||||
"settings.local_keys": "Not here, because each machine's server.yaml keeps them:",
|
||||
"settings.note": "Why",
|
||||
"settings.note_hint": "kept with the version, for whoever reads the history",
|
||||
"settings.save": "Check and apply",
|
||||
"settings.save_hint": "Checked as a control plane would load it before it is kept; a change made since this page was opened is not overwritten.",
|
||||
"settings.saved": "Version %d is in force; every control plane applies it within seconds.",
|
||||
"settings.rolled_back": "Version %d's settings are back in force, as version %d.",
|
||||
"settings.history": "History",
|
||||
"settings.col.version": "Version",
|
||||
"settings.col.at": "When",
|
||||
"settings.col.by": "By",
|
||||
"settings.col.note": "Why",
|
||||
"settings.in_force": "in force",
|
||||
"settings.view": "view",
|
||||
"settings.no_history": "No versions yet.",
|
||||
"settings.err.unchanged": "Nothing was changed.",
|
||||
"settings.err.invalid": "Not applied: a control plane could not load it.",
|
||||
"settings.err.changed": "Not applied: someone changed the settings since this page was opened. Your edit is below; compare it with the version now in force and apply it again.",
|
||||
"settings.err.no_version": "There is no such version."
|
||||
}
|
||||
@@ -10,5 +10,6 @@
|
||||
"nav.chat": "Sohbet",
|
||||
"nav.account": "Hesap",
|
||||
"nav.sign_out": "Çıkış yap",
|
||||
"nav.docs": "Belgeler"
|
||||
"nav.docs": "Belgeler",
|
||||
"nav.settings": "Ayarlar"
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"settings.title": "Ayarlar",
|
||||
"settings.sub": "Kümenin ayarları: tek bir makineye ait olmayan her şey; her kontrol düzleminde aynı, değiştikten saniyeler sonra yürürlükte. <code>llmgw ctl config edit</code> aynı belgeyi düzenler.",
|
||||
"settings.current": "Sürüm %d, yürürlükte",
|
||||
"settings.earlier": "Sürüm %d",
|
||||
"settings.earlier_sub": "Önceki bir sürüm; yürürlükte olan sürüm %d. Geri almak bunu yeni bir sürüm olarak kaydeder, hiçbir şey kaybolmaz.",
|
||||
"settings.rollback": "%d numaralı sürümü yeniden yürürlüğe koy",
|
||||
"settings.back_to_current": "yürürlükteki sürüme dön",
|
||||
"settings.local_keys": "Burada değil, çünkü her makinenin kendi server.yaml dosyası tutar:",
|
||||
"settings.note": "Neden",
|
||||
"settings.note_hint": "sürümle birlikte saklanır, geçmişi okuyacak kişi için",
|
||||
"settings.save": "Denetle ve uygula",
|
||||
"settings.save_hint": "Saklanmadan önce bir kontrol düzleminin yükleyeceği gibi denetlenir; bu sayfa açıldıktan sonra yapılmış bir değişikliğin üzerine yazılmaz.",
|
||||
"settings.saved": "%d numaralı sürüm yürürlükte; her kontrol düzlemi saniyeler içinde uygular.",
|
||||
"settings.rolled_back": "%d numaralı sürümün ayarları yeniden yürürlükte; yeni sürüm numarası %d.",
|
||||
"settings.history": "Geçmiş",
|
||||
"settings.col.version": "Sürüm",
|
||||
"settings.col.at": "Ne zaman",
|
||||
"settings.col.by": "Kim",
|
||||
"settings.col.note": "Neden",
|
||||
"settings.in_force": "yürürlükte",
|
||||
"settings.view": "görüntüle",
|
||||
"settings.no_history": "Henüz sürüm yok.",
|
||||
"settings.err.unchanged": "Hiçbir şey değiştirilmedi.",
|
||||
"settings.err.invalid": "Uygulanmadı: bir kontrol düzlemi bunu yükleyemezdi.",
|
||||
"settings.err.changed": "Uygulanmadı: bu sayfa açıldıktan sonra biri ayarları değiştirdi. Düzenlemeniz aşağıda; yürürlükteki sürümle karşılaştırıp yeniden uygulayın.",
|
||||
"settings.err.no_version": "Böyle bir sürüm yok."
|
||||
}
|
||||
+11
-7
@@ -26,6 +26,9 @@ type Nav struct {
|
||||
Mirror, Quotas bool
|
||||
// Guard is whether the guardrails page was mounted.
|
||||
Guard bool
|
||||
// Settings offers the cluster's settings: mounted, and to an
|
||||
// administrator only.
|
||||
Settings bool
|
||||
// CSRF signs the sign-out form.
|
||||
CSRF string
|
||||
// Org offers the organization page: to an administrator, and to anyone
|
||||
@@ -38,13 +41,14 @@ type navKey struct{}
|
||||
// navFor builds the menu for a signed-in person.
|
||||
func (p *AccountPages) navFor(r *http.Request, user store.User, current string) Nav {
|
||||
n := Nav{
|
||||
Current: current,
|
||||
Admin: user.Staff(),
|
||||
Role: user.StaffRole(),
|
||||
Mirror: p.hasMirror,
|
||||
Quotas: p.hasQuotas,
|
||||
Guard: p.hasGuard,
|
||||
CSRF: csrfFor(r),
|
||||
Current: current,
|
||||
Admin: user.Staff(),
|
||||
Role: user.StaffRole(),
|
||||
Mirror: p.hasMirror,
|
||||
Quotas: p.hasQuotas,
|
||||
Guard: p.hasGuard,
|
||||
Settings: p.hasSettings && user.IsAdmin,
|
||||
CSRF: csrfFor(r),
|
||||
}
|
||||
// Only when there is an organization to show: a link that leads to "there
|
||||
// are none" on every page is noise.
|
||||
|
||||
@@ -905,3 +905,13 @@ a.sel-link.small { font-size: .78rem; color: var(--muted); margin-left: .4rem; }
|
||||
/* The language switcher at the foot of every page. */
|
||||
.langs a { color: var(--muted); }
|
||||
.langs b { font-weight: 600; }
|
||||
|
||||
/* The cluster's settings, edited as the YAML document they are. */
|
||||
.settings-form textarea, pre.settings-body {
|
||||
width: 100%; box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||
font-size: .85rem; line-height: 1.4; padding: .6rem .7rem; border: 1px solid var(--line);
|
||||
border-radius: 6px; background: var(--card); color: var(--fg); tab-size: 2; overflow-x: auto;
|
||||
}
|
||||
.settings-form label { display: block; margin: .8rem 0; }
|
||||
.settings-form input[type=text] { width: 100%; box-sizing: border-box; }
|
||||
.sr-only { position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); white-space: nowrap; }
|
||||
@@ -0,0 +1,50 @@
|
||||
{{define "admin_settings"}}<!doctype html>
|
||||
<html lang="{{lang}}"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>{{t "settings.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>{{t "settings.title"}}</h1>
|
||||
<p class="sub">{{th "settings.sub"}}</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
{{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
|
||||
{{if .Notice}}<p class="ok" role="status">{{.Notice}}</p>{{end}}
|
||||
|
||||
{{if .Earlier}}
|
||||
<h2>{{t "settings.earlier" .Shown.Version}}</h2>
|
||||
<p class="sub">{{t "settings.earlier_sub" .Current.Version}}</p>
|
||||
<pre class="settings-body">{{.Body}}</pre>
|
||||
<form method="post" action="/admin/settings/rollback">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<input type="hidden" name="version" value="{{.Shown.Version}}">
|
||||
<button type="submit">{{t "settings.rollback" .Shown.Version}}</button>
|
||||
<a href="/admin/settings">{{t "settings.back_to_current"}}</a>
|
||||
</form>
|
||||
{{else}}
|
||||
<h2>{{t "settings.current" .Current.Version}}</h2>
|
||||
{{if .LocalKeys}}<p class="sub">{{t "settings.local_keys"}} {{range $i, $k := .LocalKeys}}{{if $i}}, {{end}}<code>{{$k}}</code>{{end}}.</p>{{end}}
|
||||
<form method="post" action="/admin/settings" class="settings-form">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<input type="hidden" name="base" value="{{.Current.Version}}">
|
||||
<label for="settings-body" class="sr-only">{{t "settings.title"}}</label>
|
||||
<textarea id="settings-body" name="body" rows="30" spellcheck="false" class="mono">{{.Body}}</textarea>
|
||||
<label>{{t "settings.note"}} <span class="opt">{{t "settings.note_hint"}}</span>
|
||||
<input type="text" name="note" value="{{.Note}}" maxlength="200"></label>
|
||||
<button type="submit">{{t "settings.save"}}</button>
|
||||
<p class="sub">{{t "settings.save_hint"}}</p>
|
||||
</form>
|
||||
{{end}}
|
||||
|
||||
<h2>{{t "settings.history"}}</h2>
|
||||
{{if .History}}
|
||||
<table class="rows">
|
||||
<tr><th class="num">{{t "settings.col.version"}}</th><th>{{t "settings.col.at"}}</th><th>{{t "settings.col.by"}}</th><th>{{t "settings.col.note"}}</th><th></th></tr>
|
||||
{{range .History}}<tr{{if .Selected}} aria-current="true"{{end}}>
|
||||
<td class="num">{{.Version}}</td><td>{{.At}}</td><td>{{.By}}</td><td>{{.Note}}</td>
|
||||
<td>{{if .InForce}}<strong>{{t "settings.in_force"}}</strong>{{else}}<a href="/admin/settings?version={{.Version}}">{{t "settings.view"}}</a>{{end}}</td>
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
{{else}}<p class="sub">{{t "settings.no_history"}}</p>{{end}}
|
||||
<footer>llmgw {{.Version}}</footer>
|
||||
</div></body></html>{{end}}
|
||||
@@ -20,6 +20,9 @@
|
||||
{{- if .Guard}}
|
||||
<a href="/admin/guard"{{if eq .Current "guard"}} aria-current="page"{{end}}>{{t "nav.guard"}}</a>
|
||||
{{- end}}
|
||||
{{- if .Settings}}
|
||||
<a href="/admin/settings"{{if eq .Current "settings"}} aria-current="page"{{end}}>{{t "nav.settings"}}</a>
|
||||
{{- end}}
|
||||
{{- /* An organization is a panel of the people page for an administrator;
|
||||
/org is the managers' view, and leads an administrator there. */}}
|
||||
<span class="menu-gap" aria-hidden="true"></span>
|
||||
|
||||
Reference in new issue
Block a user