Watch the cluster and edit every setting from the console

The console gains live views of nodes, GPUs, models, replicas, usage and
agent logs, and a settings tree over every key of server.yaml (agent.yaml on
a node), generated from the configuration structs: values with their
defaults and doc comments, edited in dialogs, saved only once the service's
own loader accepts them, comments kept. The generator now tells blocks apart
by package, which brings the TLS, engine and supervisor settings back into
the reference.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-03 22:54:09 +02:00
1 parent aa3efcebbe
commit 9ab93a0aa6
13 files changed
+2702 -472

No files matched your search

+29
View File
@@ -75,6 +75,35 @@ and a figure the driver does not give is left out rather than shown as zero.
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
### The console watches the cluster and holds every setting
`sudo llmgw` on a control plane is a console in three groups down the left:
- **Watch**, refreshed every three seconds: the overview; nodes, with CPU,
memory, GPUs free, disk and agent version, and a node's detail — load,
engines, each card's busy, memory, temperature and power; every GPU in the
fleet and which model holds it; models, with a model's profiles, warnings
and replicas; replicas and where they run; the last day's use by team,
project or organization; and a node's agent log.
- **Settings**: every key of `server.yaml`, block by block, with what applies
when the file sets nothing and what each key is for, from its comment in
the code. Enter changes a key or opens a block; lists and maps of blocks
(endpoints, models, prices) take entries added and removed. A save is read
by the same loader the service uses first — nothing is written that would
not load — keeps the file's comments, leaves the old one beside it, and
offers to restart the service.
- **Do**: add a node, replicate the store, leave.
On a node, `sudo llmgw` opens the same settings for its `agent.yaml`.
The keys come from the configuration structs themselves, through the
generator that writes `docs/reference.md`, so a setting added to the code is on
the screen without anyone listing it, and a test fails if the two drift.
That generator told blocks apart by name alone, so `tlsconf.Config`,
`engines.Config` and `supervisor.Config` were each taken for the agent's
`Config` and dropped: the reference had no TLS, engine or supervisor settings
at all. It now has 37 blocks, six more than before.
### A second control plane with `llmgw server add`, and the screens as windows
- **`sudo llmgw server add [user@]host [--vip 10.0.0.160/24]`**, run on a
+395 -197
View File
@@ -39,31 +39,9 @@ rather than repeated.
| `flush_interval` | `time.Duration` | [Auth](#auth) |
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
| `name` | `string` | [Band](#band) |
| `priority` | `int` | [Band](#band) |
| `saturation_ceiling` | `float64` | [Band](#band) |
| `tools` | `bool` | [Capabilities](#capabilities) |
| `vision` | `bool` | [Capabilities](#capabilities) |
| `embeddings` | `bool` | [Capabilities](#capabilities) |
| `node_id` | `string` | [Config](#config) |
| `server` | `string` | [Config](#config) |
| `servers` | `[]string` | [Config](#config) |
| `hostname` | `string` | [Config](#config) |
| `mgmt_addr` | `string` | [Config](#config) |
| `labels` | `map[string]string` | [Config](#config) |
| `mgmt_interface` | `string` | [Config](#config) |
| `fabrics` | `[]inventory.FabricAttachment` | [Config](#config) |
| `heartbeat_interval` | `time.Duration` | [Config](#config) |
| `reconnect_base` | `time.Duration` | [Config](#config) |
| `reconnect_max` | `time.Duration` | [Config](#config) |
| `adopt_ready_timeout` | `time.Duration` | [Config](#config) |
| `tls` | `tlsconf.Config` | [Config](#config) |
| `model_dirs` | `[]string` | [Config](#config) |
| `engines_dir` | `string` | [Config](#config) |
| `weights_cache` | `string` | [Config](#config) |
| `discovery` | `discovery.Options` | [Config](#config) |
| `engines` | `engines.Config` | [Config](#config) |
| `supervisor` | `supervisor.Config` | [Config](#config) |
| `interval` | `time.Duration` | [Control](#control) |
| `base_backoff` | `time.Duration` | [Control](#control) |
| `max_backoff` | `time.Duration` | [Control](#control) |
@@ -132,8 +110,6 @@ rather than repeated.
| `match_ratio` | `float64` | [Prefix](#prefix) |
| `balance_abs` | `float64` | [Prefix](#prefix) |
| `balance_rel` | `float64` | [Prefix](#prefix) |
| `prompt_per_1k` | `int64` | [Price](#price) |
| `completion_per_1k` | `int64` | [Price](#price) |
| `name` | `string` | [Profile](#profile) |
| `engine` | `string` | [Profile](#profile) |
| `gpu_class` | `string` | [Profile](#profile) |
@@ -208,6 +184,56 @@ rather than repeated.
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
| `name` | `string` | [admission.Band](#admissionband) |
| `priority` | `int` | [admission.Band](#admissionband) |
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
| `default_band` | `string` | [admission.Config](#admissionconfig) |
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
| `queue_capacity_per_band` | `int` | [admission.Config](#admissionconfig) |
| `queue_capacity_total` | `int` | [admission.Config](#admissionconfig) |
| `interval` | `time.Duration` | [admission.Config](#admissionconfig) |
| `node_id` | `string` | [agent.Config](#agentconfig) |
| `server` | `string` | [agent.Config](#agentconfig) |
| `servers` | `[]string` | [agent.Config](#agentconfig) |
| `hostname` | `string` | [agent.Config](#agentconfig) |
| `mgmt_addr` | `string` | [agent.Config](#agentconfig) |
| `labels` | `map[string]string` | [agent.Config](#agentconfig) |
| `mgmt_interface` | `string` | [agent.Config](#agentconfig) |
| `fabrics` | `[]inventory.FabricAttachment` | [agent.Config](#agentconfig) |
| `heartbeat_interval` | `time.Duration` | [agent.Config](#agentconfig) |
| `reconnect_base` | `time.Duration` | [agent.Config](#agentconfig) |
| `reconnect_max` | `time.Duration` | [agent.Config](#agentconfig) |
| `adopt_ready_timeout` | `time.Duration` | [agent.Config](#agentconfig) |
| `tls` | `tlsconf.Config` | [agent.Config](#agentconfig) |
| `model_dirs` | `[]string` | [agent.Config](#agentconfig) |
| `engines_dir` | `string` | [agent.Config](#agentconfig) |
| `weights_cache` | `string` | [agent.Config](#agentconfig) |
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `insecure` | `bool` | [tlsconf.Config](#tlsconfconfig) |
## Adapter
@@ -347,7 +373,7 @@ table costs whatever DefaultPrice says, which is nothing unless set.
`Price`
Takes a [Price](#price) block.
Takes a [config.Price](#configprice) block.
### `flush_interval`
@@ -367,20 +393,6 @@ BufferSize bounds the queue. Beyond it records are dropped rather than
blocking a request: losing accounting is bad, turning a stalled database
into an outage is worse.
## Band
### `name`
`string`
### `priority`
`int`
### `saturation_ceiling`
`float64`
## Capabilities
### `tools`
@@ -395,152 +407,6 @@ into an outage is worse.
`bool`
## Config
### `node_id`
`string`
NodeID defaults to the hostname. It must equal the common name of the
agent's certificate, so set it only where the two differ.
### `server`
`string`
Server is the control plane, and Servers is the same thing when there is
more than one. Both are accepted: every configuration written before
there could be a second control plane names one address, and none of
them should have to change.
The order is tried in order, and a refusal moves to the next. Only the
leader accepts an agent, so a standby is skipped in the time it takes to
be told so — which is what makes failover a reconnect rather than an
edit to every node's configuration.
### `servers`
`[]string`
### `hostname`
`string`
Hostname defaults to the machine's.
### `mgmt_addr`
`string`
MgmtAddr is the address engines are reached on. It defaults to this
machine's address on the route to the control plane, which is the one
the control plane and the router can reach.
### `labels`
`map[string]string`
Labels are free-form facts about a node for a profile's node_selector.
Nothing needs them: memory, GPUs, engines and disk are measured, and a
label restating one of those goes stale the day the machine changes.
### `mgmt_interface`
`string`
MgmtInterface is the ethernet interface NCCL bootstraps over. It
defaults to the interface holding MgmtAddr, then eth0.
### `fabrics`
`[]inventory.FabricAttachment`
Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
default subnet prefix is the same on every fabric, so nothing can be
inferred from the wire (plan §5.2).
### `heartbeat_interval`
`time.Duration`
### `reconnect_base`
`time.Duration`
ReconnectBase is the first reconnect delay; it backs off from there.
### `reconnect_max`
`time.Duration`
### `adopt_ready_timeout`
`time.Duration`
AdoptReadyTimeout bounds how long an adopted engine is given to answer a
health check before the agent stops waiting on it.
Shorter than a startup timeout on purpose: this engine is already
running, so it is not loading a model, it is either serving or it is not.
### `tls`
`tlsconf.Config`
TLS authenticates both ends of the control-plane stream. Leaving it unset
keeps the plaintext behaviour, which only belongs on a trusted network.
Takes a [Config](#config) block.
### `model_dirs`
`[]string`
ModelDirs are the directories this node keeps model weights in, beyond
WeightsCache, which is always looked in. Storage differs at every installation,
so llmgw observes it rather than owning it: a Lustre mount, an image
exported over the fabric, local NVMe and a copy made by hand all look
identical from here, and placement only needs to know whether the file
is present.
### `engines_dir`
`string`
EnginesDir is where the agent installs engines, one directory per
build. Defaults to /opt/llmgw/engines.
### `weights_cache`
`string`
WeightsCache is where a remote source is downloaded to. Defaults to the
first entry of ModelDirs, because a node that says where its models live
has already answered this question, and otherwise to
/var/lib/llmgw-weights.
Fetching is the simple arrangement: every node keeps its own copy. The
exotic ones need no code — weights on a shared filer or an image exported
over the fabric are simply present, and present means nothing is fetched.
### `discovery`
`discovery.Options`
Takes a [Options](#options) block.
### `engines`
`engines.Config`
Takes a [Config](#config) block.
### `supervisor`
`supervisor.Config`
Takes a [Config](#config) block.
## Control
### `interval`
@@ -769,7 +635,7 @@ agents use. Empty leaves the connection in plain HTTP, which is only
acceptable on a network where nothing untrusted can reach that port —
the mirror says as much in its own log when it starts that way.
Takes a [Config](#config) block.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `registry`
@@ -1031,16 +897,6 @@ before affinity is abandoned.
`float64`
## Price
### `prompt_per_1k`
`int64`
### `completion_per_1k`
`int64`
## Profile
### `name`
@@ -1402,7 +1258,7 @@ AgentTLS authenticates the fleet-facing port. It deliberately does not
cover the client-facing one: that surface is fronted by a publicly
trusted certificate, while this one is mutual and private.
Takes a [Config](#config) block.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `monitoring`
@@ -1594,3 +1450,345 @@ node and has not been checked against a real cluster.
RayBin is used when LaunchMode is "ray".
## accounting.Price
### `prompt_per_1k`
`int64`
### `completion_per_1k`
`int64`
## admission.Band
### `name`
`string`
### `priority`
`int`
Priority orders bands; higher is served first.
### `saturation_ceiling`
`float64`
SaturationCeiling is the pool saturation above which this band waits.
1.0 means "never hold this band back".
## admission.Config
### `bands`
`[]Band`
### `default_band`
`string`
### `queue_ttl`
`time.Duration`
QueueTTL drops a request that has waited this long. Sending the GPU work
the client has already given up on helps nobody.
### `queue_capacity_per_band`
`int`
QueueCapacityPerBand bounds each band's queue. Rejecting immediately
beats accumulating silently.
### `queue_capacity_total`
`int`
QueueCapacityTotal bounds everything queued, across bands. When it is
reached, a request is not refused while a lower band has requests
queued: the newest of the lowest band is turned away instead
(displacement), since the oldest have waited longest and are nearest
their TTL. Zero takes half of every band's capacity together.
### `interval`
`time.Duration`
Interval is how often the dispatcher re-checks saturation. Releases also
wake it, so this is only the backstop for saturation falling on its own.
## agent.Config
### `node_id`
`string`
NodeID defaults to the hostname. It must equal the common name of the
agent's certificate, so set it only where the two differ.
### `server`
`string`
Server is the control plane, and Servers is the same thing when there is
more than one. Both are accepted: every configuration written before
there could be a second control plane names one address, and none of
them should have to change.
The order is tried in order, and a refusal moves to the next. Only the
leader accepts an agent, so a standby is skipped in the time it takes to
be told so — which is what makes failover a reconnect rather than an
edit to every node's configuration.
### `servers`
`[]string`
### `hostname`
`string`
Hostname defaults to the machine's.
### `mgmt_addr`
`string`
MgmtAddr is the address engines are reached on. It defaults to this
machine's address on the route to the control plane, which is the one
the control plane and the router can reach.
### `labels`
`map[string]string`
Labels are free-form facts about a node for a profile's node_selector.
Nothing needs them: memory, GPUs, engines and disk are measured, and a
label restating one of those goes stale the day the machine changes.
### `mgmt_interface`
`string`
MgmtInterface is the ethernet interface NCCL bootstraps over. It
defaults to the interface holding MgmtAddr, then eth0.
### `fabrics`
`[]inventory.FabricAttachment`
Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
default subnet prefix is the same on every fabric, so nothing can be
inferred from the wire (plan §5.2).
### `heartbeat_interval`
`time.Duration`
### `reconnect_base`
`time.Duration`
ReconnectBase is the first reconnect delay; it backs off from there.
### `reconnect_max`
`time.Duration`
### `adopt_ready_timeout`
`time.Duration`
AdoptReadyTimeout bounds how long an adopted engine is given to answer a
health check before the agent stops waiting on it.
Shorter than a startup timeout on purpose: this engine is already
running, so it is not loading a model, it is either serving or it is not.
### `tls`
`tlsconf.Config`
TLS authenticates both ends of the control-plane stream. Leaving it unset
keeps the plaintext behaviour, which only belongs on a trusted network.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `model_dirs`
`[]string`
ModelDirs are the directories this node keeps model weights in, beyond
WeightsCache, which is always looked in. Storage differs at every installation,
so llmgw observes it rather than owning it: a Lustre mount, an image
exported over the fabric, local NVMe and a copy made by hand all look
identical from here, and placement only needs to know whether the file
is present.
### `engines_dir`
`string`
EnginesDir is where the agent installs engines, one directory per
build. Defaults to /opt/llmgw/engines.
### `weights_cache`
`string`
WeightsCache is where a remote source is downloaded to. Defaults to the
first entry of ModelDirs, because a node that says where its models live
has already answered this question, and otherwise to
/var/lib/llmgw-weights.
Fetching is the simple arrangement: every node keeps its own copy. The
exotic ones need no code — weights on a shared filer or an image exported
over the fabric are simply present, and present means nothing is fetched.
### `discovery`
`discovery.Options`
Takes a [Options](#options) block.
### `engines`
`engines.Config`
Takes a [engines.Config](#enginesconfig) block.
### `supervisor`
`supervisor.Config`
Takes a [supervisor.Config](#supervisorconfig) block.
## config.Band
### `name`
`string`
### `priority`
`int`
### `saturation_ceiling`
`float64`
## config.Price
### `prompt_per_1k`
`int64`
### `completion_per_1k`
`int64`
## engines.Config
### `disabled`
`[]string`
Disabled are engines this node never installs or advertises, whatever
the control plane asks: vLLM on a node of cards no build of it runs on,
that is meant for llama.cpp alone.
### `vllm`
`VLLMConfig`
Takes a [VLLMConfig](#vllmconfig) block.
### `sglang`
`SGLangConfig`
Takes a [SGLangConfig](#sglangconfig) block.
### `llamacpp`
`LlamaCppConfig`
Takes a [LlamaCppConfig](#llamacppconfig) block.
### `sim`
`SimConfig`
Takes a [SimConfig](#simconfig) block.
## supervisor.Config
### `work_dir`
`string`
WorkDir holds pid files and per-shard scratch space.
### `log_lines`
`int`
LogLines is how many lines of engine output to retain per shard.
### `stop_timeout`
`time.Duration`
StopTimeout is how long a process gets to exit after SIGTERM before it is
killed. Engines need time to free GPU memory cleanly.
### `health_interval`
`time.Duration`
HealthInterval is how often a starting shard is polled.
### `rendezvous_timeout`
`time.Duration`
RendezvousTimeout bounds the wait for a multi-node rendezvous point to
start listening.
### `alive_grace`
`time.Duration`
AliveGrace is how long a non-serving rank must stay up before it counts
as ready. There is no external signal for a worker, so staying alive is
the only evidence available.
## tlsconf.Config
### `ca`
`string`
CA is the bundle used to verify the other side. In a real deployment this
is whichever authority issued the fleet's certificates.
### `cert`
`string`
Cert and Key are this component's own identity.
### `key`
`string`
### `insecure`
`bool`
Insecure disables TLS entirely. It exists for local runs against
sim-engines; anything using it must stay on a trusted network.
File diff suppressed because it is too large. Load diff
+53
View File
@@ -24,6 +24,8 @@ type navItem struct {
Current bool
// Dim greys an entry that cannot be chosen now.
Dim bool
// Heading is a title between groups of entries, not an entry.
Heading bool
}
type frame struct {
@@ -70,6 +72,10 @@ func (f frame) render() string {
for _, it := range f.Nav {
mark := " "
label := it.Label
if it.Heading {
nav.WriteString(sDim.Render(label) + "\n")
continue
}
switch {
case it.State == "done":
mark = sOK.Render("✓ ")
@@ -182,3 +188,50 @@ func truncateWidth(s string, w int) string {
}
return ansi.Truncate(s, max(w-1, 0), "…")
}
// table draws rows under a header in columns as wide as they need, cut to
// width; the selected row is shown reversed when the panel has the focus.
func table(headers []string, rows [][]string, selected int, width int, focused bool) string {
widths := make([]int, len(headers))
for i, h := range headers {
widths[i] = ansi.StringWidth(h)
}
for _, r := range rows {
for i := range headers {
if i < len(r) {
widths[i] = min(max(widths[i], ansi.StringWidth(r[i])), 44)
}
}
}
line := func(cells []string) string {
var b strings.Builder
for i := range headers {
c := ""
if i < len(cells) {
c = truncateWidth(cells[i], widths[i])
}
b.WriteString(c)
if i < len(headers)-1 {
b.WriteString(strings.Repeat(" ", widths[i]-ansi.StringWidth(c)+2))
}
}
return truncateWidth(b.String(), width)
}
var b strings.Builder
b.WriteString(" " + lipgloss.NewStyle().Bold(true).Foreground(tuiDim).Render(line(headers)) + "\n")
for i, r := range rows {
l := line(r)
switch {
case i == selected && focused:
pad := max(0, width-ansi.StringWidth(ansi.Strip(l)))
l = fBar.Render(ansi.Strip(l) + strings.Repeat(" ", pad))
case i == selected:
l = lipgloss.NewStyle().Bold(true).Render(l)
}
b.WriteString(" " + l + "\n")
}
if len(rows) == 0 {
b.WriteString(" " + sDim.Render("nothing yet") + "\n")
}
return b.String()
}
+4 -224
View File
@@ -1,23 +1,13 @@
package cli
import (
"bufio"
"bytes"
"context"
"errors"
"fmt"
"io"
"os"
"runtime"
"strings"
"time"
tea "github.com/charmbracelet/bubbletea"
"github.com/mattn/go-isatty"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
)
// A bare `llmgw` in a terminal: the setup screens on a machine with nothing
@@ -43,11 +33,11 @@ func runHome(cmd *cobra.Command) error {
// that stopped halfway has written server.yaml and nothing that runs, and
// the setup screens are what finish it.
case exists("/etc/llmgw/server.yaml") && exists("/etc/systemd/system/llmgw-server.service"):
return runConsole(cmd.Context(), out)
return runConsole(cmd.Context(), out, "/etc/llmgw/server.yaml")
case exists("/etc/llmgw/agent.yaml"):
fmt.Fprintln(out, "This machine is a node. Its cluster is managed from the control plane:")
fmt.Fprintln(out, "run llmgw there. Here, `llmgw doctor` checks the machine.")
return nil
// A node: its own settings. The cluster is watched from the
// control plane.
return runConsole(cmd.Context(), out, "/etc/llmgw/agent.yaml")
}
return runSetupTUI(cmd.Context(), out)
}
@@ -56,213 +46,3 @@ func exists(p string) bool {
_, err := os.Stat(p)
return err == nil
}
// What the console asks to be done once its screen closes: each runs its own
// screen, and the console comes back after.
type consoleAction struct {
kind string // "", "node", "patroni"
targets []string
}
type summaryMsg struct {
text string
store bool
err error
}
type consoleItem struct {
label, command, ask string
kind string
n int // how many targets to ask for
}
type consoleModel struct {
client *ctlClient
summary string
store bool
err error
focus int
asking bool
asked []string
prompt textField
action consoleAction
width int
height int
spin spinnerModel
loaded bool
}
func (m consoleModel) items() []consoleItem {
its := []consoleItem{{label: "Add a node", kind: "node", n: 1,
ask: "user@address of the machine", command: "sudo llmgw node add <user>@<address>"}}
if !m.store {
its = append(its, consoleItem{label: "Replicate the store across two machines (Patroni)", kind: "patroni", n: 2,
ask: "user@address of database machine",
command: "sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node --no-model"})
}
return append(its, consoleItem{label: "Leave", kind: "quit"})
}
func (m consoleModel) fetch() tea.Msg {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
var s adminapi.Summary
if err := m.client.get(ctx, "/api/v1/summary", &s); err != nil {
return summaryMsg{err: err}
}
var b bytes.Buffer
c := &cobra.Command{}
c.SetOut(&b)
printSummary(c, s)
return summaryMsg{text: b.String(), store: len(s.Store) > 0}
}
func (m consoleModel) Init() tea.Cmd { return tea.Batch(m.fetch, m.spin.Tick) }
func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.WindowSizeMsg:
m.width, m.height = msg.Width, msg.Height
case spinTickMsg:
var cmd tea.Cmd
m.spin, cmd = m.spin.Update(msg)
return m, cmd
case summaryMsg:
m.loaded = true
m.err = msg.err
if msg.err == nil {
m.summary, m.store = msg.text, msg.store
}
return m, tea.Tick(5*time.Second, func(time.Time) tea.Msg { return m.fetch() })
case tea.KeyMsg:
if msg.Type == tea.KeyCtrlC {
return m, tea.Quit
}
its := m.items()
if m.asking {
switch msg.Type {
case tea.KeyEsc:
m.asking, m.asked = false, nil
case tea.KeyEnter:
v := strings.TrimSpace(m.prompt.Value())
if v == "" {
return m, nil
}
m.asked = append(m.asked, v)
m.prompt.SetValue("")
if it := its[m.focus]; len(m.asked) == it.n {
m.action = consoleAction{kind: it.kind, targets: m.asked}
return m, tea.Quit
}
default:
var cmd tea.Cmd
m.prompt, cmd = m.prompt.Update(msg)
return m, cmd
}
return m, nil
}
switch msg.String() {
case "up", "k", "shift+tab":
m.focus = (m.focus - 1 + len(its)) % len(its)
case "down", "j", "tab":
m.focus = (m.focus + 1) % len(its)
case "q":
return m, tea.Quit
case "enter":
it := its[m.focus]
if it.kind == "quit" {
return m, tea.Quit
}
m.asking, m.asked = true, nil
m.prompt = newTextField()
m.prompt.Placeholder, m.prompt.Width = "ubuntu@10.0.0.20", 40
m.prompt.Focus()
}
}
return m, nil
}
func (m consoleModel) View() string {
host, _ := os.Hostname()
var nav []navItem
for i, it := range m.items() {
nav = append(nav, navItem{Label: it.label, Current: i == m.focus})
}
var body string
switch {
case !m.loaded:
body = "\n " + m.spin.View() + " asking the control plane…"
case m.err != nil:
body = "\n " + sBad.Render("The control plane does not answer:") + "\n " + m.err.Error() +
"\n\n " + sDim.Render("systemctl status llmgw-server says why.")
default:
body = "\n" + indent(m.summary, " ")
}
if it := m.items()[m.focus]; it.command != "" {
body += "\n\n " + sDim.Render("On the command line:") + "\n " + it.command
}
f := frame{
Width: m.width, Height: m.height,
Title: "llmgw console", Where: host,
NavTitle: "Do", Nav: nav,
PanelTitle: "Cluster", Body: body,
Keys: "↑↓ choose · Enter do it · Q leave · refreshed every five seconds",
}
if m.asking {
it := m.items()[m.focus]
ask := it.ask
if it.n > 1 {
ask = fmt.Sprintf("%s %d of %d", ask, len(m.asked)+1, it.n)
}
f.DialogTitle = it.label
f.Dialog = ask + "\n\n" + m.prompt.View() + "\n\n" + sDim.Render("Enter · Esc cancel")
}
return f.render()
}
func indent(s, pre string) string {
lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
for i := range lines {
lines[i] = pre + lines[i]
}
return strings.Join(lines, "\n")
}
// runConsole shows the console, runs what is chosen on it, and comes back.
func runConsole(ctx context.Context, out io.Writer) error {
addr := "127.0.0.1:9090"
if cfg, err := config.Load("/etc/llmgw/server.yaml"); err == nil && cfg.AdminListen != "" {
addr = cfg.AdminListen
if strings.HasPrefix(addr, ":") {
addr = "127.0.0.1" + addr
}
}
for {
m := consoleModel{client: newCtlClient(addr), spin: newSpinner(), width: 100, height: 30}
final, err := tea.NewProgram(m, tea.WithOutput(out), tea.WithAltScreen()).Run()
if err != nil {
return err
}
act := final.(consoleModel).action
switch act.kind {
case "":
return nil
case "node":
c := newNodeAddCmd()
c.SetArgs(act.targets)
c.SetOut(out)
err = c.ExecuteContext(ctx)
case "patroni":
in := newDefaultServerInit()
in.patroni = act.targets
// Already set up: only the store changes.
in.noMirror, in.noNode, in.noModel = true, true, true
err = in.runInitTUI(ctx, out)
}
if err != nil {
fmt.Fprintln(out, sBad.Render("✗ ")+err.Error())
}
fmt.Fprint(out, "\n"+sDim.Render("enter to go back to the console")+" ")
_, _ = bufio.NewReader(os.Stdin).ReadString('\n')
}
}
+54 -19
View File
@@ -1,6 +1,8 @@
package cli
import (
"os"
"path/filepath"
"strings"
"testing"
@@ -104,35 +106,68 @@ func TestTheChecksHoldTheSetupBackOnlyForAFailure(t *testing.T) {
func TestTheConsoleOffersToReplicateOnlyAStoreThatIsNot(t *testing.T) {
m := consoleModel{}
if !strings.Contains(m.items()[1].label, "Replicate") {
if !m.entryOK(cReplicate) {
t.Error("a single store is not offered replication")
}
m.store = true
for _, it := range m.items() {
if strings.Contains(it.label, "Replicate") {
t.Error("a replicated store is offered replication")
}
m.data.sum.Store = []adminapi.StoreMember{{Name: "a", Role: "leader"}}
if m.entryOK(cReplicate) {
t.Error("a replicated store is offered replication")
}
// A node is asked for, then the console closes with it to do.
var mm tea.Model = consoleModel{summary: "x", loaded: true}
// A node is asked for in a dialog, then the console closes with it to do.
var mm tea.Model = consoleModel{menu: cAddNode, loaded: true, width: 100, height: 30}
mm = press(mm, "enter", "ubuntu@10.0.0.9", "enter")
if a := mm.(consoleModel).action; a.kind != "node" || a.targets[0] != "ubuntu@10.0.0.9" {
t.Errorf("%+v", a)
}
_ = adminapi.Summary{}
}
func TestTheSetupIsDrawnAsWindows(t *testing.T) {
m := newSetupModel("ctl", []string{"10.0.0.5"})
m.width, m.height = 100, 30
v := m.View()
lines := strings.Split(v, "\n")
if len(lines) != 30 {
t.Errorf("%d lines for a 30-line terminal", len(lines))
func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "server.yaml")
if err := os.WriteFile(file, []byte("# mine\nlisten: \":8080\"\n"), 0o644); err != nil {
t.Fatal(err)
}
for _, want := range []string{"llmgw setup", "Steps", "Start", "╭", "One machine", "Ctrl+C"} {
if !strings.Contains(v, want) {
t.Errorf("the screen lacks %q:\n%s", want, v)
m := newConsoleModel("127.0.0.1:1", file)
m.width, m.height, m.loaded = 120, 40, true
m.data.nodes = []adminapi.NodeView{{ID: "cuda01", State: "ready", GPUTotal: 1, GPUs: []adminapi.GPUView{{Index: 0, Name: "Quadro RTX 6000"}}}}
m.data.replicas = []adminapi.ReplicaView{{ID: "r1", Model: "coder-14b", State: "ready",
Shards: []adminapi.ShardView{{Node: "cuda01", GPUs: []int{0}}}}}
// Watching: the GPUs view says which model holds each card.
m.menu = cGPUs
if v := m.View(); !strings.Contains(v, "Quadro RTX 6000") || !strings.Contains(v, "coder-14b") {
t.Errorf("the GPUs view:\n%s", v)
}
// Settings: into router, change retries, save.
var mm tea.Model = m
mm.(consoleModel).View()
cm := mm.(consoleModel)
cm.menu, cm.inPanel = cSettings, true
for i, r := range cm.settingsRows() {
if r.label == "router" {
cm.row = i
}
}
mm = press(cm, "enter") // into router
cm = mm.(consoleModel)
for i, r := range cm.settingsRows() {
if r.label == "retries" {
cm.row = i
}
}
if v := cm.View(); !strings.Contains(v, "(default)") {
t.Errorf("an unset key does not show its default:\n%s", v)
}
mm = press(cm, "enter") // the edit dialog
cm = mm.(consoleModel)
cm.prompt.SetValue("5")
mm = press(cm, "enter", "s")
if d := mm.(consoleModel).dialog; d != "saved" {
t.Fatalf("after saving the dialog is %q: %s", d, mm.(consoleModel).dialogText)
}
b, _ := os.ReadFile(file)
if !strings.Contains(string(b), "retries: 5") || !strings.Contains(string(b), "# mine") {
t.Errorf("server.yaml:\n%s", b)
}
}
+51 -21
View File
@@ -29,19 +29,30 @@ type Field struct {
Key string
Type string
Doc string
// Nested names the struct this key expands into, empty for a leaf. It is
// what lets the reference say "see Auth" rather than printing the same
// twenty keys under four parents.
// Nested names the struct this key expands into, by its ID
// (config.Auth), empty for a leaf. It is what lets the reference say
// "see Auth" rather than printing the same twenty keys under four
// parents.
Nested string
}
// Block is a struct that appears in configuration.
type Block struct {
Name string
// Package and Name are where the struct is declared: what tells
// tlsconf.Config from the agent's Config.
Package string
Name string
// Title is what it is called in the reference and on the screens: the
// name alone where no other block shares it, the package's name with it
// where one does.
Title string
Doc string
Fields []Field
}
// ID is the block's package and name, unique where Name is not.
func (b Block) ID() string { return b.Package + "." + b.Name }
// Collect reads koanf-tagged structs out of Go source files.
func Collect(files []string) ([]Block, error) {
fset := token.NewFileSet()
@@ -62,22 +73,36 @@ func Collect(files []string) ([]Block, error) {
if !ok {
return true
}
b, ok := blockOf(ts, st)
if !ok || seen[b.Name] {
b, ok := blockOf(f.Name.Name, ts, st)
if !ok || seen[b.ID()] {
return true
}
seen[b.Name] = true
seen[b.ID()] = true
blocks = append(blocks, b)
return true
})
}
sort.Slice(blocks, func(i, j int) bool { return blocks[i].Name < blocks[j].Name })
// Named by their own name unless another block has it too. Keyed by
// name alone, tlsconf.Config, engines.Config and supervisor.Config were
// each taken for the agent's Config and dropped: the reference had no
// TLS settings at all.
count := map[string]int{}
for _, b := range blocks {
count[b.Name]++
}
for i := range blocks {
blocks[i].Title = blocks[i].Name
if count[blocks[i].Name] > 1 {
blocks[i].Title = blocks[i].ID()
}
}
sort.Slice(blocks, func(i, j int) bool { return blocks[i].Title < blocks[j].Title })
return blocks, nil
}
// blockOf builds a Block, or reports that this struct carries no settings.
func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
b := Block{Name: ts.Name.Name, Doc: docText(ts.Doc)}
func blockOf(pkg string, ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
b := Block{Package: pkg, Name: ts.Name.Name, Doc: docText(ts.Doc)}
for _, f := range st.Fields.List {
if f.Tag == nil {
continue
@@ -90,7 +115,7 @@ func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
if !ok || key == "" || key == "-" {
continue
}
typ, nested := typeOf(f.Type)
typ, nested := typeOf(pkg, f.Type)
b.Fields = append(b.Fields, Field{
Key: key, Type: typ, Doc: docText(f.Doc), Nested: nested,
})
@@ -99,29 +124,34 @@ func blockOf(ts *ast.TypeSpec, st *ast.StructType) (Block, bool) {
}
// typeOf renders a field's type, and names the struct it expands into.
func typeOf(e ast.Expr) (rendered, nested string) {
func typeOf(pkg string, e ast.Expr) (rendered, nested string) {
switch t := e.(type) {
case *ast.Ident:
if isNamedStruct(t.Name) {
return t.Name, t.Name
return t.Name, pkg + "." + t.Name
}
return t.Name, ""
case *ast.SelectorExpr:
// A type from another package, such as tlsconf.Config.
// A type from another package, such as tlsconf.Config. Standard
// library types are values, not blocks: a time.Duration is a
// setting, written "30s".
name := t.Sel.Name
if pkg, ok := t.X.(*ast.Ident); ok {
return pkg.Name + "." + name, name
if x, ok := t.X.(*ast.Ident); ok {
if x.Name == "time" {
return x.Name + "." + name, ""
}
return x.Name + "." + name, x.Name + "." + name
}
return name, name
return name, ""
case *ast.ArrayType:
inner, _ := typeOf(t.Elt)
inner, _ := typeOf(pkg, t.Elt)
return "[]" + inner, ""
case *ast.MapType:
k, _ := typeOf(t.Key)
v, _ := typeOf(t.Value)
k, _ := typeOf(pkg, t.Key)
v, _ := typeOf(pkg, t.Value)
return "map[" + k + "]" + v, ""
case *ast.StarExpr:
inner, n := typeOf(t.X)
inner, n := typeOf(pkg, t.X)
return "*" + inner, n
default:
return "value", ""
+15
View File
@@ -136,3 +136,18 @@ func TestAFieldMarkedNotConfigurableIsLeftOut(t *testing.T) {
}
}
}
func TestTheSettingsScreensDescribeTheCode(t *testing.T) {
// The screens list Reference; it must be what the structs say now.
blocks, err := Collect(sourcePaths(t))
if err != nil {
t.Fatal(err)
}
have, err := os.ReadFile(filepath.Join(repoRoot(t), "internal", "docgen", "reference_gen.go"))
if err != nil {
t.Fatalf("reference_gen.go is missing; run go generate ./...: %v", err)
}
if string(have) != RenderGo(blocks) {
t.Error("internal/docgen/reference_gen.go no longer matches the configuration structs; run go generate ./...")
}
}
+5 -1
View File
@@ -38,5 +38,9 @@ func main() {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
fmt.Printf("docs/reference.md: %d blocks\n", len(blocks))
if err := os.WriteFile(filepath.Join(root, "internal", "docgen", "reference_gen.go"), []byte(docgen.RenderGo(blocks)), 0o644); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
fmt.Printf("docs/reference.md and internal/docgen/reference_gen.go: %d blocks\n", len(blocks))
}
+300
View File
@@ -0,0 +1,300 @@
// Code generated by go run ./internal/docgen/gen; DO NOT EDIT.
package docgen
// Reference is every configuration block, as Collect reads them.
var Reference = []Block{
{Package: "catalog", Name: "Adapter", Title: "Adapter", Doc: "", Fields: []Field{
{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
{Key: "source", Type: "string", Doc: "Source is the adapter file (GGUF for llama.cpp), fetched by each node\nas the model's weights are.", Nested: ""},
{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
}},
{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
{Key: "default_band", Type: "string", Doc: "", Nested: ""},
{Key: "band_header", Type: "string", Doc: "BandHeader lets a client declare its band. Until authentication lands in\nv1.0.0 this is trusted, so the port must not be exposed to callers who\nshould not be able to promote themselves.", Nested: ""},
{Key: "queue_ttl", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "queue_capacity_per_band", Type: "int", Doc: "", Nested: ""},
{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued across bands. Reached, a\nrequest is not refused while a lower band has any queued: the newest of\nthe lowest band is turned away instead, with a 503 and Retry-After.\nZero takes half of all the bands' capacities together.", Nested: ""},
{Key: "interval", Type: "time.Duration", Doc: "", Nested: ""},
}},
{Package: "config", Name: "Auth", Title: "Auth", Doc: "", Fields: []Field{
{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
{Key: "store", Type: "string", Doc: "Store is the SQLite file holding keys, tenancy and usage.", Nested: ""},
{Key: "cache_ttl", Type: "time.Duration", Doc: "CacheTTL bounds how long a revoked key keeps working, traded against a\ndatabase read per key per TTL.", Nested: ""},
{Key: "negative_ttl", Type: "time.Duration", Doc: "NegativeTTL is how long a rejected key is remembered as rejected, so a\nclient looping with a bad key cannot become a database read per request.", Nested: ""},
{Key: "default_rpm", Type: "int", Doc: "DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.\n\nUnlimited used to be the default, and a project created without limits is\nthe easy path — the \"default\" project on this cluster still is one. The\nfirst sign of that is the bill or the outage. Zero keeps the old\nbehaviour for anyone who wants it, but it should be a choice rather than\nan omission. A fallback, not a ceiling: an explicit limit at any scope\nwins, above or below.", Nested: ""},
{Key: "default_tpm", Type: "int", Doc: "", Nested: ""},
{Key: "signup_open", Type: "bool", Doc: "SignupOpen serves the signup form. Accounts it creates are pending until\nan operator approves them, so this is not a way in — but turning it off\nmeans accounts are made with `llmgw users create` instead.", Nested: ""},
{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
{Key: "pricing", Type: "map[string]Price", Doc: "Pricing is what each model costs, for budgets. A model absent from the\ntable costs whatever DefaultPrice says, which is nothing unless set.", Nested: ""},
{Key: "default_price", Type: "Price", Doc: "", Nested: "config.Price"},
{Key: "flush_interval", Type: "time.Duration", Doc: "FlushInterval and BatchSize decide when usage records are written.", Nested: ""},
{Key: "batch_size", Type: "int", Doc: "", Nested: ""},
{Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""},
}},
{Package: "catalog", Name: "Capabilities", Title: "Capabilities", Doc: "", Fields: []Field{
{Key: "tools", Type: "bool", Doc: "", Nested: ""},
{Key: "vision", Type: "bool", Doc: "", Nested: ""},
{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
}},
{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
{Key: "base_backoff", Type: "time.Duration", Doc: "BaseBackoff and MaxBackoff bound how fast a failing model is retried.", Nested: ""},
{Key: "max_backoff", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "unreachable_after", Type: "time.Duration", Doc: "UnreachableAfter and LostAfter stage node expiry. They are deliberately\nfar apart: a network blip must not tear down replicas that still serve,\nand rebuilding a multi-node replica costs tens of minutes of loading.", Nested: ""},
{Key: "lost_after", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "port_range", Type: "[]int", Doc: "PortRange is the per-node range for engine HTTP and dist-init ports.", Nested: ""},
{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over.", Nested: ""},
{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
}},
{Package: "config", Name: "Endpoint", Title: "Endpoint", Doc: "", Fields: []Field{
{Key: "id", Type: "string", Doc: "", Nested: ""},
{Key: "url", Type: "string", Doc: "", Nested: ""},
{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
{Key: "models", Type: "[]string", Doc: "", Nested: ""},
{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
{Key: "pool", Type: "string", Doc: "", Nested: ""},
{Key: "weight", Type: "float64", Doc: "", Nested: ""},
{Key: "priority", Type: "int", Doc: "", Nested: ""},
}},
{Package: "config", Name: "EngineOverride", Title: "EngineOverride", Doc: "", Fields: []Field{
{Key: "gpu_class", Type: "string", Doc: "GPUClass picks nodes with a GPU of this class (\"v100-32g\").", Nested: ""},
{Key: "nodes", Type: "[]string", Doc: "Nodes picks nodes by id.", Nested: ""},
{Key: "labels", Type: "map[string]string", Doc: "Labels picks nodes carrying all of these.", Nested: ""},
{Key: "engines", Type: "map[string]string", Doc: "Engines are laid over the cluster's: a version, or \"\" for none.", Nested: ""},
}},
{Package: "catalog", Name: "External", Title: "External", Doc: "", Fields: []Field{
{Key: "url", Type: "string", Doc: "URL is the provider's base address, without /v1: https://api.openai.com.", Nested: ""},
{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
}},
{Package: "config", Name: "Landing", Title: "Landing", Doc: "", Fields: []Field{
{Key: "contact", Type: "string", Doc: "Contact is where someone without a key should go — an address, a URL, a\nchannel name. Rendered as a link when it is one, as text otherwise.", Nested: ""},
{Key: "notice", Type: "string", Doc: "Notice is an optional line of your own, for whatever the reader needs to\nknow that this page cannot guess.", Nested: ""},
}},
{Package: "engines", Name: "LlamaCppConfig", Title: "LlamaCppConfig", Doc: "", Fields: []Field{
{Key: "bin", Type: "string", Doc: "Bin is the llama-server binary. It is the server, not the CLI: the\ngateway talks OpenAI over HTTP and needs the one that serves it.\nEmpty means the agent installs llama.cpp itself, at the version the\ncontrol plane asks for. Setting it opts out of that.", Nested: ""},
{Key: "parallel", Type: "int", Doc: "Parallel is how many requests a server handles at once, when a profile\ndoes not say. llama.cpp defaults to one, which would make every second\nrequest wait behind the first no matter what the router decided.", Nested: ""},
{Key: "threads", Type: "int", Doc: "Threads is how many CPU threads a shard with no GPU computes on. Zero\ntakes half this machine's CPUs.\n\nNot all of them, which is llama.cpp's own default: two busy servers\nthat each spin on every CPU fight for them, and on an 8-vCPU node\nthat made 7B generation fall from 10 tokens a second to under 1 each.\nAt half, one server alone is about a third slower and two together\nkeep 5.5 each. A node that only ever runs one CPU replica does better\nwith all of them; say so here.", Nested: ""},
}},
{Package: "config", Name: "Mirror", Title: "Mirror", Doc: "", Fields: []Field{
{Key: "api", Type: "string", Doc: "API is the mirror's management address, scheme included, for example\nhttps://models.internal:8082. Empty means no mirror is configured.\n\nThe management port, not the one nodes fetch bytes from. Those are two\nlisteners on the mirror precisely so this one can be locked down without\nre-certifying every agent's fetcher.", Nested: ""},
{Key: "tls", Type: "tlsconf.Config", Doc: "TLS authenticates to the mirror's management API, with the same CA the\nagents use. Empty leaves the connection in plain HTTP, which is only\nacceptable on a network where nothing untrusted can reach that port —\nthe mirror says as much in its own log when it starts that way.", Nested: "tlsconf.Config"},
{Key: "registry", Type: "string", Doc: "Registry is the model registry the mirror page browses, empty for the\npublic Hugging Face. Set it to an internal mirror of the registry where\nthe control plane cannot reach the internet — or leave the whole mirror\nblock out, which turns the explorer off rather than giving it a search\nbox that can only time out.", Nested: ""},
{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds a management call. It does not bound a download: the\nmirror answers a fetch as soon as it has started one, and the bytes keep\narriving long after this deadline has passed.", Nested: ""},
}},
{Package: "catalog", Name: "Model", Title: "Model", Doc: "", Fields: []Field{
{Key: "name", Type: "string", Doc: "", Nested: ""},
{Key: "aliases", Type: "[]string", Doc: "", Nested: ""},
{Key: "description", Type: "string", Doc: "Description is what this model is for, in a line.\n\nWritten by whoever added it, because nobody else can say it. A registry\noffers a pipeline tag and a wall of README, and neither answers the\nquestion a person picking from a list is asking: which of these nine\nnames should I send this request to.\n\nIt reaches clients through /v1/models, which is the only view most of\nthem ever see. A gateway serving nine models by name alone makes that\nchoice by guesswork.", Nested: ""},
{Key: "source", Type: "string", Doc: "Source is the weights path or a HuggingFace id.", Nested: ""},
{Key: "profiles", Type: "[]Profile", Doc: "", Nested: ""},
{Key: "scaling", Type: "Scaling", Doc: "", Nested: "catalog.Scaling"},
{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it.", Nested: ""},
{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
{Key: "routing_profile", Type: "string", Doc: "RoutingProfile selects how the router scores endpoints for this model.", Nested: ""},
{Key: "slo", Type: "SLO", Doc: "", Nested: "catalog.SLO"},
{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
{Key: "max_output", Type: "int", Doc: "", Nested: ""},
{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
}},
{Package: "config", Name: "Monitoring", Title: "Monitoring", Doc: "", Fields: []Field{
{Key: "prometheus", Type: "string", Doc: "Prometheus is the server's address, for example\nhttp://monitoring.internal:9090. Empty leaves the charts off the\ndashboard rather than showing an error where one should be.", Nested: ""},
{Key: "window", Type: "time.Duration", Doc: "Window is how far back the charts look. Zero takes an hour.", Nested: ""},
{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds one query. It runs while somebody waits for a page, so a\nmonitoring system that is slow today is skipped rather than waited for.", Nested: ""},
}},
{Package: "discovery", Name: "Options", Title: "Options", Doc: "", Fields: []Field{
{Key: "nvidia_smi", Type: "string", Doc: "NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.\nWorth configuring on nodes where the driver lives outside the default\npaths, which containerized driver installs often do.", Nested: ""},
{Key: "ib_root", Type: "string", Doc: "IBRoot is the sysfs InfiniBand directory.", Nested: ""},
{Key: "prefer_nvml", Type: "bool", Doc: "PreferNVML uses the NVML backend when this binary has one. NVLink\ntopology is read from nvidia-smi topo -m on every build; NVML, where it\nis used, is the more exact source of the peers.", Nested: ""},
{Key: "skip_gpus", Type: "bool", Doc: "SkipGPUs omits GPU discovery. A node without nvidia-smi is already\nfound to have none, so this is for one whose driver is broken.", Nested: ""},
}},
{Package: "config", Name: "Personal", Title: "Personal", Doc: "", Fields: []Field{
{Key: "disabled", Type: "bool", Doc: "Disabled refuses personal keys and personal chat; people then use the\ngateway only through projects they are members of.", Nested: ""},
{Key: "rpm", Type: "int", Doc: "", Nested: ""},
{Key: "tpm", Type: "int", Doc: "", Nested: ""},
{Key: "budget", Type: "float64", Doc: "Budget is in whole units per Period (\"day\" or \"month\").", Nested: ""},
{Key: "period", Type: "string", Doc: "", Nested: ""},
}},
{Package: "config", Name: "Prefix", Title: "Prefix", Doc: "", Fields: []Field{
{Key: "enabled", Type: "bool", Doc: "", Nested: ""},
{Key: "max_nodes", Type: "int", Doc: "MaxNodes bounds the tree, MinMatch is the shortest match worth acting on\nin bytes, and TTL is how long an entry is believed — an engine evicts on\nits own schedule, so an old record may describe a cache that is gone.", Nested: ""},
{Key: "min_match", Type: "int", Doc: "", Nested: ""},
{Key: "ttl", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "match_ratio", Type: "float64", Doc: "MatchRatio is the share of the prompt that must match to be worth acting\non. BalanceAbs and BalanceRel are the imbalance guard, and both must trip\nbefore affinity is abandoned.", Nested: ""},
{Key: "balance_abs", Type: "float64", Doc: "", Nested: ""},
{Key: "balance_rel", Type: "float64", Doc: "", Nested: ""},
}},
{Package: "catalog", Name: "Profile", Title: "Profile", Doc: "", Fields: []Field{
{Key: "name", Type: "string", Doc: "", Nested: ""},
{Key: "engine", Type: "string", Doc: "", Nested: ""},
{Key: "gpu_class", Type: "string", Doc: "", Nested: ""},
{Key: "tp", Type: "int", Doc: "", Nested: ""},
{Key: "pp", Type: "int", Doc: "", Nested: ""},
{Key: "gpus_per_node", Type: "int", Doc: "GPUsPerNode defaults to TP for a single-node profile.", Nested: ""},
{Key: "nodes", Type: "int", Doc: "Nodes above 1 makes this a multi-node profile, which requires a fabric.", Nested: ""},
{Key: "args", Type: "[]string", Doc: "", Nested: ""},
{Key: "env", Type: "map[string]string", Doc: "", Nested: ""},
{Key: "min_memory_gb", Type: "int", Doc: "MinMemoryGB is the host RAM this profile needs to run.\n\nThe number that matters is the model's working set: the weights, plus the\nKV cache for the whole context pool, plus the engine's own overhead. For\na GPU profile that lives in the cards and this is small; for a CPU\nprofile it is the whole thing, and getting it wrong means an OOM on a\nnode that reported itself healthy.\n\nZero means unconstrained, which is every profile that existed before\nthis. A node that does not report its memory is not excluded either —\nan older agent must not have its nodes quietly emptied by an upgrade of\nthe control plane.", Nested: ""},
{Key: "cpu_cores", Type: "int", Doc: "CPUCores is the cores a replica of a profile with no GPU is given:\nreserved out of its node's CPUs by placement, as memory is, and what\nits engine computes on (llama.cpp's --threads).\n\nReserved because an engine takes every core it is told about, and two\non one node that each think they have all of them spin against each\nother: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second\nto under one each. Zero reserves nothing and leaves the thread count to\nthe node, which is every profile from before this.", Nested: ""},
{Key: "min_disk_gb", Type: "int", Doc: "MinDiskGB is the free space a node needs before it can hold these\nweights.\n\nOnly checked when the node does not already have them: a node holding\nthe file needs no room for it, and requiring the space twice would keep\na model off the one node best placed to run it.\n\nDeclared rather than derived, because the size of a model that has never\nbeen fetched anywhere is not knowable from here — and a wrong guess\nfails late, halfway through a download, which is the failure this exists\nto prevent. Zero means unconstrained, and a node that reports no storage\nis not excluded: unknown is not full.", Nested: ""},
{Key: "node_selector", Type: "map[string]string", Doc: "NodeSelector restricts this profile to nodes carrying every one of these\nlabels, with these values.\n\nA CPU profile otherwise fits everywhere, because there is no card to be\nfree — which is right for a 0.5B model and wrong for a 14B one. Nothing\nin the inventory says how much memory a node has, so the operator says\nwhich nodes are meant for a model instead of the scheduler inferring it.\nAgents have reported labels since the beginning; this is what reads them.", Nested: ""},
{Key: "weight", Type: "float64", Doc: "Weight is the router's capacity weight. It should come from ctl bench\nrather than a guess (plan §6.2.1).", Nested: ""},
{Key: "priority", Type: "int", Doc: "Priority orders profiles during placement; highest that fits wins.", Nested: ""},
{Key: "startup_timeout", Type: "time.Duration", Doc: "StartupTimeout bounds model loading. A 70B model across two nodes takes\ntens of minutes, so this is per profile rather than global.", Nested: ""},
}},
{Package: "config", Name: "Router", Title: "Router", Doc: "", Fields: []Field{
{Key: "activation_wait", Type: "time.Duration", Doc: "ActivationWait is how long a request waits for an on-demand model to\nload before the client is told to come back. Zero takes the default.\n\nThe one real decision in on-demand loading. Refusing outright makes the\nfeature useless, since the first request for every model would fail;\nwaiting without a bound turns a request into a job, and a client that\nbudgeted thirty seconds ends up holding a connection for ten minutes.", Nested: ""},
{Key: "scrape_interval", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "scrape_timeout", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "stale_after", Type: "time.Duration", Doc: "StaleAfter is how long an endpoint may go unscraped before being scored\ndown (plan §7.6).", Nested: ""},
{Key: "retries", Type: "int", Doc: "Retries is how many *additional* endpoints may be tried after a failure,\nand only while nothing has been written to the client (plan §7.7).", Nested: ""},
{Key: "cooldown", Type: "time.Duration", Doc: "Cooldown is how long a failing endpoint leaves the candidate pool.", Nested: ""},
{Key: "failure_threshold", Type: "int", Doc: "FailureThreshold is the consecutive failure count that opens a breaker.\n1 turns the breaker into a pure cooldown.", Nested: ""},
{Key: "default_profile", Type: "string", Doc: "DefaultProfile names the routing profile used when a model picks none.", Nested: ""},
{Key: "prefix", Type: "Prefix", Doc: "Prefix turns on the approximate prefix tree (plan §7.5 tier 1). Off, the\nrouter uses session affinity alone, which is tier 0 and already most of\nthe win for multi-turn traffic.", Nested: "config.Prefix"},
{Key: "session_header", Type: "string", Doc: "SessionHeader carries the session id for affinity (plan §7.5 tier 0).", Nested: ""},
{Key: "inject_usage", Type: "bool", Doc: "InjectUsage asks engines for usage even when the client did not, so token\naccounting works regardless of client behaviour.", Nested: ""},
{Key: "max_body_bytes", Type: "int64", Doc: "MaxBodyBytes bounds the request body the router will read.", Nested: ""},
{Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"},
{Key: "scoring", Type: "Scoring", Doc: "", Nested: "config.Scoring"},
{Key: "admission", Type: "Admission", Doc: "", Nested: "config.Admission"},
}},
{Package: "engines", Name: "SGLangConfig", Title: "SGLangConfig", Doc: "", Fields: []Field{
{Key: "python", Type: "string", Doc: "Python is the interpreter of the SGLang virtualenv. SGLang is launched as\na module rather than a console script, which is the documented entry\npoint and avoids depending on the script being on PATH.", Nested: ""},
}},
{Package: "catalog", Name: "SLO", Title: "SLO", Doc: "", Fields: []Field{
{Key: "ttft_p99", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "tpot_p99", Type: "time.Duration", Doc: "", Nested: ""},
}},
{Package: "catalog", Name: "Scaling", Title: "Scaling", Doc: "", Fields: []Field{
{Key: "min", Type: "int", Doc: "", Nested: ""},
{Key: "max", Type: "int", Doc: "", Nested: ""},
{Key: "idle_timeout", Type: "time.Duration", Doc: "IdleTimeout is how long an on-demand model may go unasked-for before its\nreplicas are stopped. Zero takes the cluster default.\n\nMinutes rather than seconds, always. Stopping a replica throws away the\nKV cache prefix routing spent its effort filling, and a model that flaps\nin and out loses that entirely — so the saving has to be worth more than\nthe cache, which it only is over a real idle period.", Nested: ""},
}},
{Package: "config", Name: "Scoring", Title: "Scoring", Doc: "", Fields: []Field{
{Key: "target_waiting", Type: "float64", Doc: "", Nested: ""},
{Key: "target_inflight", Type: "float64", Doc: "", Nested: ""},
{Key: "kv_ceiling", Type: "float64", Doc: "", Nested: ""},
{Key: "session_load_factor", Type: "float64", Doc: "", Nested: ""},
}},
{Package: "config", Name: "Server", Title: "Server", Doc: "", Fields: []Field{
{Key: "listen", Type: "string", Doc: "Listen is the client-facing OpenAI API address.", Nested: ""},
{Key: "admin_listen", Type: "string", Doc: "AdminListen serves the dashboard, the management API, health and pprof.\n\nLoopback by default, and it should stay there: pprof can dump the process\nheap, and API keys are in that heap in plaintext. Every comment beside\nthis setting has always said not to expose it, while the default bound\nevery interface — a default that contradicts its own documentation is a\nfootgun, so the default now matches the advice.", Nested: ""},
{Key: "metrics_listen", Type: "string", Doc: "MetricsListen serves Prometheus metrics and nothing else.\n\nSeparate from AdminListen because the reason that one is loopback is\npprof, not metrics — and metrics are the one thing on it that something\nelse genuinely needs to reach. Sharing a listener meant a fleet could\neither be monitored or keep its heap private, which is not a choice\nanybody should have to make.\n\nEmpty means off, and metrics remain available on AdminListen either way,\nso nothing that scrapes it today has to change.", Nested: ""},
{Key: "agent_listen", Type: "string", Doc: "AgentListen is where node agents connect. Separate from the client port\nso the fleet-facing surface can be firewalled independently.", Nested: ""},
{Key: "auth", Type: "Auth", Doc: "Auth turns on API keys on the client-facing port.", Nested: "config.Auth"},
{Key: "threads", Type: "Threads", Doc: "Threads keeps conversations and agent memory without API keys, for a\ngateway on one trusted machine. With auth on they live in the auth store,\nper project, and this block is ignored.", Nested: "config.Threads"},
{Key: "landing", Type: "Landing", Doc: "Landing configures the page served at \"/\".", Nested: "config.Landing"},
{Key: "agent_tls", Type: "tlsconf.Config", Doc: "AgentTLS authenticates the fleet-facing port. It deliberately does not\ncover the client-facing one: that surface is fronted by a publicly\ntrusted certificate, while this one is mutual and private.", Nested: "tlsconf.Config"},
{Key: "monitoring", Type: "Monitoring", Doc: "Monitoring is where the dashboard reads its history from.", Nested: "config.Monitoring"},
{Key: "tracing", Type: "Tracing", Doc: "Tracing sends a trace of each inference request to an OpenTelemetry\ncollector. Off unless an endpoint is set here or in\nOTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "config.Tracing"},
{Key: "mirror", Type: "Mirror", Doc: "Mirror points the control plane at a model mirror it can administer.\nEmpty means there is none, and /admin/mirror stays unmounted.", Nested: "config.Mirror"},
{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
{Key: "models", Type: "[]catalog.Model", Doc: "", Nested: ""},
{Key: "endpoints", Type: "[]Endpoint", Doc: "Endpoints is the phase-1 static replica list. It is only consulted when\nno model declares a profile, which is what makes a router-only smoke test\nagainst sim-engines possible without a control plane.", Nested: ""},
}},
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
}},
{Package: "config", Name: "Threads", Title: "Threads", Doc: "", Fields: []Field{
{Key: "store", Type: "string", Doc: "Store is the SQLite file. Empty means nothing is kept.", Nested: ""},
{Key: "retain_hours", Type: "int", Doc: "RetainHours is how long idle conversations and notes are kept. Required\nwith Store: a store with no expiry keeps everything anyone typed, by\nomission.", Nested: ""},
}},
{Package: "config", Name: "Timeouts", Title: "Timeouts", Doc: "", Fields: []Field{
{Key: "dial", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "response_header", Type: "time.Duration", Doc: "ResponseHeader bounds the wait for headers on a streaming request. It is\na liveness check there, because headers arrive before generation starts.", Nested: ""},
{Key: "generation", Type: "time.Duration", Doc: "Generation bounds a non-streaming request, where the engine sends no\nheaders until the completion is done. Setting response_header alone used\nto cap both, so 60s read as a liveness guard and behaved as a limit of\nabout a hundred tokens on a slow model.", Nested: ""},
{Key: "first_token", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "idle", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "total", Type: "time.Duration", Doc: "", Nested: ""},
}},
{Package: "config", Name: "Tracing", Title: "Tracing", Doc: "", Fields: []Field{
{Key: "endpoint", Type: "string", Doc: "Endpoint is the collector, for example http://otel.internal:4318.\nEmpty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.", Nested: ""},
{Key: "sample_ratio", Type: "float64", Doc: "SampleRatio is the share of requests traced, 0 to 1, when the caller\ndid not decide it with a sampled traceparent of its own. Zero takes 1.", Nested: ""},
}},
{Package: "engines", Name: "VLLMConfig", Title: "VLLMConfig", Doc: "", Fields: []Field{
{Key: "bin", Type: "string", Doc: "Bin is the vllm executable.", Nested: ""},
{Key: "launch_mode", Type: "string", Doc: "LaunchMode selects the multi-node mechanism: \"mp\", the default, or\n\"ray\".\n\nmp is what vLLM documents today — every node runs serve with its rank,\nthe workers headless — and needs nothing but vLLM. ray is the older way,\nkept for a vLLM too old for mp's multi-node flags; it needs Ray on every\nnode and has not been checked against a real cluster.", Nested: ""},
{Key: "ray_bin", Type: "string", Doc: "RayBin is used when LaunchMode is \"ray\".", Nested: ""},
}},
{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
}},
{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
{Key: "name", Type: "string", Doc: "", Nested: ""},
{Key: "priority", Type: "int", Doc: "Priority orders bands; higher is served first.", Nested: ""},
{Key: "saturation_ceiling", Type: "float64", Doc: "SaturationCeiling is the pool saturation above which this band waits.\n1.0 means \"never hold this band back\".", Nested: ""},
}},
{Package: "admission", Name: "Config", Title: "admission.Config", Doc: "", Fields: []Field{
{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
{Key: "default_band", Type: "string", Doc: "", Nested: ""},
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL drops a request that has waited this long. Sending the GPU work\nthe client has already given up on helps nobody.", Nested: ""},
{Key: "queue_capacity_per_band", Type: "int", Doc: "QueueCapacityPerBand bounds each band's queue. Rejecting immediately\nbeats accumulating silently.", Nested: ""},
{Key: "queue_capacity_total", Type: "int", Doc: "QueueCapacityTotal bounds everything queued, across bands. When it is\nreached, a request is not refused while a lower band has requests\nqueued: the newest of the lowest band is turned away instead\n(displacement), since the oldest have waited longest and are nearest\ntheir TTL. Zero takes half of every band's capacity together.", Nested: ""},
{Key: "interval", Type: "time.Duration", Doc: "Interval is how often the dispatcher re-checks saturation. Releases also\nwake it, so this is only the backstop for saturation falling on its own.", Nested: ""},
}},
{Package: "agent", Name: "Config", Title: "agent.Config", Doc: "", Fields: []Field{
{Key: "node_id", Type: "string", Doc: "NodeID defaults to the hostname. It must equal the common name of the\nagent's certificate, so set it only where the two differ.", Nested: ""},
{Key: "server", Type: "string", Doc: "Server is the control plane, and Servers is the same thing when there is\nmore than one. Both are accepted: every configuration written before\nthere could be a second control plane names one address, and none of\nthem should have to change.\n\nThe order is tried in order, and a refusal moves to the next. Only the\nleader accepts an agent, so a standby is skipped in the time it takes to\nbe told so — which is what makes failover a reconnect rather than an\nedit to every node's configuration.", Nested: ""},
{Key: "servers", Type: "[]string", Doc: "", Nested: ""},
{Key: "hostname", Type: "string", Doc: "Hostname defaults to the machine's.", Nested: ""},
{Key: "mgmt_addr", Type: "string", Doc: "MgmtAddr is the address engines are reached on. It defaults to this\nmachine's address on the route to the control plane, which is the one\nthe control plane and the router can reach.", Nested: ""},
{Key: "labels", Type: "map[string]string", Doc: "Labels are free-form facts about a node for a profile's node_selector.\nNothing needs them: memory, GPUs, engines and disk are measured, and a\nlabel restating one of those goes stale the day the machine changes.", Nested: ""},
{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over. It\ndefaults to the interface holding MgmtAddr, then eth0.", Nested: ""},
{Key: "fabrics", Type: "[]inventory.FabricAttachment", Doc: "Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's\ndefault subnet prefix is the same on every fabric, so nothing can be\ninferred from the wire (plan §5.2).", Nested: ""},
{Key: "heartbeat_interval", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "reconnect_base", Type: "time.Duration", Doc: "ReconnectBase is the first reconnect delay; it backs off from there.", Nested: ""},
{Key: "reconnect_max", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "adopt_ready_timeout", Type: "time.Duration", Doc: "AdoptReadyTimeout bounds how long an adopted engine is given to answer a\nhealth check before the agent stops waiting on it.\n\nShorter than a startup timeout on purpose: this engine is already\nrunning, so it is not loading a model, it is either serving or it is not.", Nested: ""},
{Key: "tls", Type: "tlsconf.Config", Doc: "TLS authenticates both ends of the control-plane stream. Leaving it unset\nkeeps the plaintext behaviour, which only belongs on a trusted network.", Nested: "tlsconf.Config"},
{Key: "model_dirs", Type: "[]string", Doc: "ModelDirs are the directories this node keeps model weights in, beyond\nWeightsCache, which is always looked in. Storage differs at every installation,\nso llmgw observes it rather than owning it: a Lustre mount, an image\nexported over the fabric, local NVMe and a copy made by hand all look\nidentical from here, and placement only needs to know whether the file\nis present.", Nested: ""},
{Key: "engines_dir", Type: "string", Doc: "EnginesDir is where the agent installs engines, one directory per\nbuild. Defaults to /opt/llmgw/engines.", Nested: ""},
{Key: "weights_cache", Type: "string", Doc: "WeightsCache is where a remote source is downloaded to. Defaults to the\nfirst entry of ModelDirs, because a node that says where its models live\nhas already answered this question, and otherwise to\n/var/lib/llmgw-weights.\n\nFetching is the simple arrangement: every node keeps its own copy. The\nexotic ones need no code — weights on a shared filer or an image exported\nover the fabric are simply present, and present means nothing is fetched.", Nested: ""},
{Key: "discovery", Type: "discovery.Options", Doc: "", Nested: "discovery.Options"},
{Key: "engines", Type: "engines.Config", Doc: "", Nested: "engines.Config"},
{Key: "supervisor", Type: "supervisor.Config", Doc: "", Nested: "supervisor.Config"},
}},
{Package: "config", Name: "Band", Title: "config.Band", Doc: "", Fields: []Field{
{Key: "name", Type: "string", Doc: "", Nested: ""},
{Key: "priority", Type: "int", Doc: "", Nested: ""},
{Key: "saturation_ceiling", Type: "float64", Doc: "", Nested: ""},
}},
{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
}},
{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
{Key: "vllm", Type: "VLLMConfig", Doc: "", Nested: "engines.VLLMConfig"},
{Key: "sglang", Type: "SGLangConfig", Doc: "", Nested: "engines.SGLangConfig"},
{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
}},
{Package: "supervisor", Name: "Config", Title: "supervisor.Config", Doc: "", Fields: []Field{
{Key: "work_dir", Type: "string", Doc: "WorkDir holds pid files and per-shard scratch space.", Nested: ""},
{Key: "log_lines", Type: "int", Doc: "LogLines is how many lines of engine output to retain per shard.", Nested: ""},
{Key: "stop_timeout", Type: "time.Duration", Doc: "StopTimeout is how long a process gets to exit after SIGTERM before it is\nkilled. Engines need time to free GPU memory cleanly.", Nested: ""},
{Key: "health_interval", Type: "time.Duration", Doc: "HealthInterval is how often a starting shard is polled.", Nested: ""},
{Key: "rendezvous_timeout", Type: "time.Duration", Doc: "RendezvousTimeout bounds the wait for a multi-node rendezvous point to\nstart listening.", Nested: ""},
{Key: "alive_grace", Type: "time.Duration", Doc: "AliveGrace is how long a non-serving rank must stay up before it counts\nas ready. There is no external signal for a worker, so staying alive is\nthe only evidence available.", Nested: ""},
}},
{Package: "tlsconf", Name: "Config", Title: "tlsconf.Config", Doc: "", Fields: []Field{
{Key: "ca", Type: "string", Doc: "CA is the bundle used to verify the other side. In a real deployment this\nis whichever authority issued the fleet's certificates.", Nested: ""},
{Key: "cert", Type: "string", Doc: "Cert and Key are this component's own identity.", Nested: ""},
{Key: "key", Type: "string", Doc: "", Nested: ""},
{Key: "insecure", Type: "bool", Doc: "Insecure disables TLS entirely. It exists for local runs against\nsim-engines; anything using it must stay on a trusted network.", Nested: ""},
}},
}
+30 -10
View File
@@ -61,13 +61,13 @@ rather than repeated.
for _, bl := range blocks {
for _, f := range bl.Fields {
b.WriteString(fmt.Sprintf("| `%s` | %s | [%s](#%s) |\n",
f.Key, code(f.Type), bl.Name, anchor(bl.Name)))
f.Key, code(f.Type), bl.Title, anchor(bl.Title)))
}
}
b.WriteString("\n")
for _, bl := range blocks {
fmt.Fprintf(&b, "## %s\n\n", bl.Name)
fmt.Fprintf(&b, "## %s\n\n", bl.Title)
if bl.Doc != "" {
b.WriteString(bl.Doc + "\n\n")
}
@@ -76,24 +76,44 @@ rather than repeated.
if f.Doc != "" {
b.WriteString(f.Doc + "\n\n")
}
if f.Nested != "" && hasBlock(blocks, f.Nested) {
fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", f.Nested, anchor(f.Nested))
if nb, ok := Find(blocks, f.Nested); ok {
fmt.Fprintf(&b, "Takes a [%s](#%s) block.\n\n", nb.Title, anchor(nb.Title))
}
}
}
return b.String()
}
func hasBlock(blocks []Block, name string) bool {
// Find is the block with an ID, as a Field's Nested names it.
func Find(blocks []Block, id string) (Block, bool) {
for _, b := range blocks {
if b.Name == name {
return true
if id != "" && b.ID() == id {
return b, true
}
}
return false
return Block{}, false
}
func code(s string) string { return "`" + s + "`" }
// anchor is how GitHub and Gitea slug a heading.
func anchor(s string) string { return strings.ToLower(s) }
// anchor is how GitHub and Gitea slug a heading: lower case, and punctuation
// such as the dot in tlsconf.Config dropped.
func anchor(s string) string { return strings.ToLower(strings.ReplaceAll(s, ".", "")) }
// RenderGo writes the blocks as Go, for the settings screens: every key, its
// type and what it does, from the same source as the reference, so a setting
// added to the code is on the screens without anyone listing it.
func RenderGo(blocks []Block) string {
var b strings.Builder
b.WriteString("// Code generated by go run ./internal/docgen/gen; DO NOT EDIT.\n\npackage docgen\n\n")
b.WriteString("// Reference is every configuration block, as Collect reads them.\nvar Reference = []Block{\n")
for _, bl := range blocks {
fmt.Fprintf(&b, "\t{Package: %q, Name: %q, Title: %q, Doc: %q, Fields: []Field{\n", bl.Package, bl.Name, bl.Title, bl.Doc)
for _, f := range bl.Fields {
fmt.Fprintf(&b, "\t\t{Key: %q, Type: %q, Doc: %q, Nested: %q},\n", f.Key, f.Type, f.Doc, f.Nested)
}
b.WriteString("\t}},\n")
}
b.WriteString("}\n")
return b.String()
}
+513
View File
@@ -0,0 +1,513 @@
// Package settings edits llmgw's configuration files key by key, for the
// settings screens: what each key is comes from the generated reference, so
// every setting the code reads is there without anyone listing it, and the
// file is edited as YAML nodes, so its comments and order survive.
package settings
import (
"bytes"
"fmt"
"os"
"reflect"
"strconv"
"strings"
"time"
"go.yaml.in/yaml/v3"
"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
)
// Kind is how a key is shown and edited.
type Kind int
// The kinds of key.
const (
Text Kind = iota
Int
Float
Bool
Duration
TextList // []string, []int: one value a line
TextMap // map[string]string and the like: key = value lines
Block // a nested block: a submenu
BlockList // a list of blocks: a submenu of entries
BlockMap // a map of blocks: a submenu of named entries
)
// Key is one key of a block, as the screens need it.
type Key struct {
docgen.Field
Kind Kind
// Block is the block it opens, for Block, BlockList and BlockMap.
Block *docgen.Block
}
// Blocks finds a block by ID in the generated reference.
func Find(id string) (*docgen.Block, bool) {
for i := range docgen.Reference {
if docgen.Reference[i].ID() == id {
return &docgen.Reference[i], true
}
}
return nil, false
}
// Keys are a block's keys with their kinds.
func Keys(b *docgen.Block) []Key {
out := make([]Key, 0, len(b.Fields))
for _, f := range b.Fields {
k := Key{Field: f}
k.Kind, k.Block = kindOf(b.Package, f)
out = append(out, k)
}
return out
}
func kindOf(pkg string, f docgen.Field) (Kind, *docgen.Block) {
if nb, ok := Find(f.Nested); ok {
return Block, nb
}
t := strings.TrimPrefix(f.Type, "*")
switch t {
case "string":
return Text, nil
case "int", "int64", "uint", "uint32":
return Int, nil
case "float64", "float32":
return Float, nil
case "bool":
return Bool, nil
case "time.Duration":
return Duration, nil
}
if elem, ok := strings.CutPrefix(t, "[]"); ok {
if b, ok := Find(qualify(pkg, elem)); ok {
return BlockList, b
}
return TextList, nil
}
if strings.HasPrefix(t, "map[") {
elem := t[strings.Index(t, "]")+1:]
if b, ok := Find(qualify(pkg, elem)); ok {
return BlockMap, b
}
return TextMap, nil
}
return Text, nil
}
// qualify is a type as a block ID: Endpoint in config is config.Endpoint.
func qualify(pkg, typ string) string {
if strings.Contains(typ, ".") {
return typ
}
return pkg + "." + typ
}
// Root is the block a file is: server.yaml is config.Server, agent.yaml the
// agent's Config.
func Root(file string) (*docgen.Block, error) {
id := "config.Server"
if strings.Contains(file, "agent") {
id = "agent.Config"
}
b, ok := Find(id)
if !ok {
return nil, fmt.Errorf("no block %s in the reference", id)
}
return b, nil
}
// Doc is a configuration file, open for editing.
type Doc struct {
Path string
root *yaml.Node // the document's top mapping
doc yaml.Node
// Changed counts edits not yet saved.
Changed int
}
// Open reads a file; a missing one is an empty document.
func Open(path string) (*Doc, error) {
d := &Doc{Path: path}
b, err := os.ReadFile(path)
if err != nil && !os.IsNotExist(err) {
return nil, err
}
if len(bytes.TrimSpace(b)) > 0 {
if err := yaml.Unmarshal(b, &d.doc); err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
}
if d.doc.Kind == 0 {
d.doc = yaml.Node{Kind: yaml.DocumentNode, Content: []*yaml.Node{{Kind: yaml.MappingNode, Tag: "!!map"}}}
}
d.root = d.doc.Content[0]
if d.root.Kind != yaml.MappingNode {
return nil, fmt.Errorf("%s is not a mapping at the top", path)
}
return d, nil
}
// Bytes is the document as it would be written.
func (d *Doc) Bytes() ([]byte, error) {
var b bytes.Buffer
enc := yaml.NewEncoder(&b)
enc.SetIndent(2)
if err := enc.Encode(&d.doc); err != nil {
return nil, err
}
return b.Bytes(), enc.Close()
}
// Node is the node at a path of keys and list indexes, or nil.
func (d *Doc) Node(path []string) *yaml.Node {
n := d.root
for _, p := range path {
n = child(n, p)
if n == nil {
return nil
}
}
return n
}
func child(n *yaml.Node, key string) *yaml.Node {
switch n.Kind {
case yaml.MappingNode:
for i := 0; i+1 < len(n.Content); i += 2 {
if n.Content[i].Value == key {
return n.Content[i+1]
}
}
case yaml.SequenceNode:
if i, err := strconv.Atoi(key); err == nil && i >= 0 && i < len(n.Content) {
return n.Content[i]
}
}
return nil
}
// ensure is the node at a path, made as mappings where there is none.
func (d *Doc) ensure(path []string) *yaml.Node {
n := d.root
for _, p := range path {
c := child(n, p)
if c == nil {
if n.Kind != yaml.MappingNode {
return nil
}
c = &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
n.Content = append(n.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: p}, c)
}
n = c
}
return n
}
// Set writes one key's value: a scalar, a list of lines or a map of
// "key = value" lines, by its kind. It checks the value first.
func (d *Doc) Set(path []string, k Kind, value string) error {
if len(path) == 0 {
return fmt.Errorf("no key")
}
n, err := encode(k, value)
if err != nil {
return err
}
parent := d.ensure(path[:len(path)-1])
if parent == nil {
return fmt.Errorf("%s is not a block", strings.Join(path[:len(path)-1], "."))
}
key := path[len(path)-1]
if parent.Kind == yaml.SequenceNode {
i, _ := strconv.Atoi(key)
parent.Content[i] = n
} else if c := child(parent, key); c != nil {
// Kept in place, with whatever comment it had.
n.HeadComment, n.LineComment, n.FootComment = c.HeadComment, c.LineComment, c.FootComment
*c = *n
} else {
parent.Content = append(parent.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key}, n)
}
d.Changed++
return nil
}
// Unset removes a key, so its default applies.
func (d *Doc) Unset(path []string) {
if len(path) == 0 {
return
}
parent := d.Node(path[:len(path)-1])
if parent == nil {
return
}
key := path[len(path)-1]
switch parent.Kind {
case yaml.MappingNode:
for i := 0; i+1 < len(parent.Content); i += 2 {
if parent.Content[i].Value == key {
parent.Content = append(parent.Content[:i], parent.Content[i+2:]...)
d.Changed++
return
}
}
case yaml.SequenceNode:
if i, err := strconv.Atoi(key); err == nil && i < len(parent.Content) {
parent.Content = append(parent.Content[:i], parent.Content[i+1:]...)
d.Changed++
}
}
}
// Append adds an empty block to the list at path, and returns its index.
func (d *Doc) Append(path []string) int {
parent := d.ensure(path[:len(path)-1])
key := path[len(path)-1]
seq := child(parent, key)
if seq == nil || seq.Kind != yaml.SequenceNode {
seq = &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
if c := child(parent, key); c != nil {
*c = *seq
seq = c
} else {
parent.Content = append(parent.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: key}, seq)
}
}
seq.Content = append(seq.Content, &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"})
d.Changed++
return len(seq.Content) - 1
}
// AddEntry adds an empty block under a name to the map at path.
func (d *Doc) AddEntry(path []string, name string) error {
m := d.ensure(path)
if m == nil || m.Kind != yaml.MappingNode {
return fmt.Errorf("%s is not a map", strings.Join(path, "."))
}
if child(m, name) != nil {
return fmt.Errorf("%s is there already", name)
}
m.Content = append(m.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: name},
&yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"})
d.Changed++
return nil
}
// Len is how many entries a list or map at path has.
func (d *Doc) Len(path []string) int {
n := d.Node(path)
switch {
case n == nil:
return 0
case n.Kind == yaml.SequenceNode:
return len(n.Content)
case n.Kind == yaml.MappingNode:
return len(n.Content) / 2
}
return 0
}
// MapKeys are a map's keys, in the file's order.
func (d *Doc) MapKeys(path []string) []string {
n := d.Node(path)
if n == nil || n.Kind != yaml.MappingNode {
return nil
}
var out []string
for i := 0; i+1 < len(n.Content); i += 2 {
out = append(out, n.Content[i].Value)
}
return out
}
// Text is a key's value as it is edited: a scalar as written, a list one
// item a line, a map "key = value" a line. ok is false for a key the file
// does not set.
func (d *Doc) Text(path []string) (string, bool) {
n := d.Node(path)
if n == nil {
return "", false
}
switch n.Kind {
case yaml.ScalarNode:
return n.Value, true
case yaml.SequenceNode:
var l []string
for _, c := range n.Content {
l = append(l, c.Value)
}
return strings.Join(l, "\n"), true
case yaml.MappingNode:
var l []string
for i := 0; i+1 < len(n.Content); i += 2 {
l = append(l, n.Content[i].Value+" = "+n.Content[i+1].Value)
}
return strings.Join(l, "\n"), true
}
return "", true
}
func encode(k Kind, value string) (*yaml.Node, error) {
v := strings.TrimSpace(value)
scalar := func(tag string) *yaml.Node { return &yaml.Node{Kind: yaml.ScalarNode, Tag: tag, Value: v} }
switch k {
case Int:
if _, err := strconv.ParseInt(v, 10, 64); err != nil {
return nil, fmt.Errorf("%q is not a whole number", v)
}
return scalar("!!int"), nil
case Float:
if _, err := strconv.ParseFloat(v, 64); err != nil {
return nil, fmt.Errorf("%q is not a number", v)
}
return scalar("!!float"), nil
case Bool:
b, err := strconv.ParseBool(v)
if err != nil {
return nil, fmt.Errorf("%q is not true or false", v)
}
return &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!bool", Value: strconv.FormatBool(b)}, nil
case Duration:
if _, err := time.ParseDuration(v); err != nil {
return nil, fmt.Errorf("%q is not a duration, like 30s, 5m or 2h", v)
}
return scalar("!!str"), nil
case TextList:
seq := &yaml.Node{Kind: yaml.SequenceNode, Tag: "!!seq"}
for _, l := range splitLines(value) {
seq.Content = append(seq.Content, &yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: l})
}
return seq, nil
case TextMap:
m := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
for _, l := range splitLines(value) {
k, val, ok := strings.Cut(l, "=")
if !ok {
return nil, fmt.Errorf("%q is not key = value", l)
}
m.Content = append(m.Content,
&yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: strings.TrimSpace(k)},
&yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: strings.TrimSpace(val)})
}
return m, nil
case Block, BlockList, BlockMap:
return nil, fmt.Errorf("a block is edited key by key")
}
return scalar("!!str"), nil
}
func splitLines(s string) []string {
var out []string
for _, l := range strings.FieldsFunc(s, func(r rune) bool { return r == '\n' || r == ',' }) {
if l = strings.TrimSpace(l); l != "" {
out = append(out, l)
}
}
return out
}
// Save checks the document with check — a loader that reads and validates
// a file — and writes it, the old file kept beside it. Nothing is written
// when the check fails.
func (d *Doc) Save(check func(path string) error) (string, error) {
b, err := d.Bytes()
if err != nil {
return "", err
}
tmp := d.Path + ".check"
if err := os.WriteFile(tmp, b, 0o600); err != nil {
return "", err
}
defer os.Remove(tmp)
if check != nil {
if err := check(tmp); err != nil {
return "", err
}
}
mode := os.FileMode(0o644)
backup := ""
if info, err := os.Stat(d.Path); err == nil {
mode = info.Mode().Perm()
old, err := os.ReadFile(d.Path)
if err != nil {
return "", err
}
backup = fmt.Sprintf("%s.bak-%s", d.Path, time.Now().Format("20060102-150405"))
if err := os.WriteFile(backup, old, mode); err != nil {
return "", err
}
}
if err := os.WriteFile(d.Path+".new", b, mode); err != nil {
return "", err
}
if err := os.Rename(d.Path+".new", d.Path); err != nil {
return "", err
}
d.Changed = 0
return backup, nil
}
// Effective is the value a loaded configuration has at a path of koanf
// keys — what applies, default included — as text.
func Effective(cfg any, path []string) (string, bool) {
v := reflect.ValueOf(cfg)
for _, p := range path {
for v.Kind() == reflect.Pointer {
if v.IsNil() {
return "", false
}
v = v.Elem()
}
switch v.Kind() {
case reflect.Struct:
found := false
t := v.Type()
for i := 0; i < t.NumField(); i++ {
if tag, _, _ := strings.Cut(t.Field(i).Tag.Get("koanf"), ","); tag == p {
v, found = v.Field(i), true
break
}
}
if !found {
return "", false
}
case reflect.Slice:
i, err := strconv.Atoi(p)
if err != nil || i >= v.Len() {
return "", false
}
v = v.Index(i)
case reflect.Map:
mv := v.MapIndex(reflect.ValueOf(p))
if !mv.IsValid() {
return "", false
}
v = mv
default:
return "", false
}
}
for v.Kind() == reflect.Pointer {
if v.IsNil() {
return "", true
}
v = v.Elem()
}
switch v.Kind() {
case reflect.Slice:
var l []string
for i := 0; i < v.Len(); i++ {
l = append(l, fmt.Sprint(v.Index(i).Interface()))
}
return strings.Join(l, ", "), true
case reflect.Struct, reflect.Map:
return "", true
}
if d, ok := v.Interface().(time.Duration); ok {
return d.String(), true
}
return fmt.Sprint(v.Interface()), true
}
+122
View File
@@ -0,0 +1,122 @@
package settings
import (
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
)
func TestEveryKeyOfBothFilesCanBeReached(t *testing.T) {
// Walked from each file's root: every block a key opens is found, so no
// setting is a dead end on the screens.
for _, file := range []string{"/etc/llmgw/server.yaml", "/etc/llmgw/agent.yaml"} {
root, err := Root(file)
if err != nil {
t.Fatal(err)
}
seen := map[string]bool{}
var walk func(b *docgen.Block, path string)
count := 0
walk = func(b *docgen.Block, path string) {
if seen[b.ID()] {
return
}
seen[b.ID()] = true
for _, k := range Keys(b) {
count++
if k.Nested != "" && k.Block == nil {
t.Errorf("%s: %s.%s opens %s, which is not in the reference", file, path, k.Key, k.Nested)
}
if k.Block != nil {
walk(k.Block, path+"."+k.Key)
}
}
}
walk(root, "")
if count < 20 {
t.Errorf("%s: only %d keys reached", file, count)
}
}
// The TLS settings, which the reference used to lose to a name clash.
root, _ := Root("server.yaml")
for _, k := range Keys(root) {
if k.Key == "agent_tls" && (k.Kind != Block || k.Block.ID() != "tlsconf.Config") {
t.Errorf("agent_tls is %v %v", k.Kind, k.Block)
}
if k.Key == "endpoints" && k.Kind != BlockList {
t.Errorf("endpoints is %v", k.Kind)
}
}
}
const sample = `# The control plane.
listen: ":8080"
router:
# How many other endpoints to try.
retries: 2
`
func TestAnEditKeepsTheFilesCommentsAndIsCheckedBeforeItIsWritten(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "server.yaml")
if err := os.WriteFile(path, []byte(sample), 0o644); err != nil {
t.Fatal(err)
}
d, err := Open(path)
if err != nil {
t.Fatal(err)
}
if err := d.Set([]string{"router", "retries"}, Int, "three"); err == nil {
t.Error("a word was taken for a number")
}
if err := d.Set([]string{"router", "retries"}, Int, "3"); err != nil {
t.Fatal(err)
}
if err := d.Set([]string{"router", "cooldown"}, Duration, "45s"); err != nil {
t.Fatal(err)
}
check := func(p string) error { _, err := config.Load(p); return err }
if _, err := d.Save(check); err != nil {
t.Fatalf("a good edit was refused: %v", err)
}
b, _ := os.ReadFile(path)
for _, want := range []string{"# The control plane.", "# How many other endpoints to try.", "retries: 3", "cooldown: 45s"} {
if !strings.Contains(string(b), want) {
t.Errorf("the file lacks %q:\n%s", want, b)
}
}
// Two endpoints with one id: the loader refuses, and nothing is written.
for range 2 {
i := d.Append([]string{"endpoints"})
for _, kv := range [][2]string{{"id", "a"}, {"url", "http://x"}} {
if err := d.Set([]string{"endpoints", itoa(i), kv[0]}, Text, kv[1]); err != nil {
t.Fatal(err)
}
}
_ = d.Set([]string{"endpoints", itoa(i), "models"}, TextList, "m")
}
if _, err := d.Save(check); err == nil {
t.Error("an invalid configuration was saved")
}
if b2, _ := os.ReadFile(path); string(b2) != string(b) {
t.Error("a refused save changed the file")
}
}
func TestTheValueThatAppliesIsShownWhenTheFileSetsNone(t *testing.T) {
cfg := config.Default()
if v, ok := Effective(cfg, []string{"router", "retries"}); !ok || v != "2" {
t.Errorf("router.retries: %q %v", v, ok)
}
if v, ok := Effective(cfg, []string{"router", "cooldown"}); !ok || v != "15s" {
t.Errorf("router.cooldown: %q", v)
}
}
func itoa(i int) string { return strconv.Itoa(i) }