Merge ha: replicated store, second control plane, join codes, setup and console screens
Release / build (push) Successful in 4m26s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 5m14s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-03 22:59:49 +02:00
commit 36cdbce125
67 files changed
+8574 -490

No files matched your search

+395 -197
View File
@@ -40,31 +40,9 @@ rather than repeated.
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
| `usage_retention` | `time.Duration` | [Auth](#auth) |
| `name` | `string` | [Band](#band) |
| `priority` | `int` | [Band](#band) |
| `saturation_ceiling` | `float64` | [Band](#band) |
| `tools` | `bool` | [Capabilities](#capabilities) |
| `vision` | `bool` | [Capabilities](#capabilities) |
| `embeddings` | `bool` | [Capabilities](#capabilities) |
| `node_id` | `string` | [Config](#config) |
| `server` | `string` | [Config](#config) |
| `servers` | `[]string` | [Config](#config) |
| `hostname` | `string` | [Config](#config) |
| `mgmt_addr` | `string` | [Config](#config) |
| `labels` | `map[string]string` | [Config](#config) |
| `mgmt_interface` | `string` | [Config](#config) |
| `fabrics` | `[]inventory.FabricAttachment` | [Config](#config) |
| `heartbeat_interval` | `time.Duration` | [Config](#config) |
| `reconnect_base` | `time.Duration` | [Config](#config) |
| `reconnect_max` | `time.Duration` | [Config](#config) |
| `adopt_ready_timeout` | `time.Duration` | [Config](#config) |
| `tls` | `tlsconf.Config` | [Config](#config) |
| `model_dirs` | `[]string` | [Config](#config) |
| `engines_dir` | `string` | [Config](#config) |
| `weights_cache` | `string` | [Config](#config) |
| `discovery` | `discovery.Options` | [Config](#config) |
| `engines` | `engines.Config` | [Config](#config) |
| `supervisor` | `supervisor.Config` | [Config](#config) |
| `interval` | `time.Duration` | [Control](#control) |
| `base_backoff` | `time.Duration` | [Control](#control) |
| `max_backoff` | `time.Duration` | [Control](#control) |
@@ -133,8 +111,6 @@ rather than repeated.
| `match_ratio` | `float64` | [Prefix](#prefix) |
| `balance_abs` | `float64` | [Prefix](#prefix) |
| `balance_rel` | `float64` | [Prefix](#prefix) |
| `prompt_per_1k` | `int64` | [Price](#price) |
| `completion_per_1k` | `int64` | [Price](#price) |
| `name` | `string` | [Profile](#profile) |
| `engine` | `string` | [Profile](#profile) |
| `gpu_class` | `string` | [Profile](#profile) |
@@ -210,6 +186,56 @@ rather than repeated.
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
| `name` | `string` | [admission.Band](#admissionband) |
| `priority` | `int` | [admission.Band](#admissionband) |
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
| `default_band` | `string` | [admission.Config](#admissionconfig) |
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
| `queue_capacity_per_band` | `int` | [admission.Config](#admissionconfig) |
| `queue_capacity_total` | `int` | [admission.Config](#admissionconfig) |
| `interval` | `time.Duration` | [admission.Config](#admissionconfig) |
| `node_id` | `string` | [agent.Config](#agentconfig) |
| `server` | `string` | [agent.Config](#agentconfig) |
| `servers` | `[]string` | [agent.Config](#agentconfig) |
| `hostname` | `string` | [agent.Config](#agentconfig) |
| `mgmt_addr` | `string` | [agent.Config](#agentconfig) |
| `labels` | `map[string]string` | [agent.Config](#agentconfig) |
| `mgmt_interface` | `string` | [agent.Config](#agentconfig) |
| `fabrics` | `[]inventory.FabricAttachment` | [agent.Config](#agentconfig) |
| `heartbeat_interval` | `time.Duration` | [agent.Config](#agentconfig) |
| `reconnect_base` | `time.Duration` | [agent.Config](#agentconfig) |
| `reconnect_max` | `time.Duration` | [agent.Config](#agentconfig) |
| `adopt_ready_timeout` | `time.Duration` | [agent.Config](#agentconfig) |
| `tls` | `tlsconf.Config` | [agent.Config](#agentconfig) |
| `model_dirs` | `[]string` | [agent.Config](#agentconfig) |
| `engines_dir` | `string` | [agent.Config](#agentconfig) |
| `weights_cache` | `string` | [agent.Config](#agentconfig) |
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `insecure` | `bool` | [tlsconf.Config](#tlsconfconfig) |
## Adapter
@@ -349,7 +375,7 @@ table costs whatever DefaultPrice says, which is nothing unless set.
`Price`
Takes a [Price](#price) block.
Takes a [config.Price](#configprice) block.
### `flush_interval`
@@ -380,20 +406,6 @@ five-minute sums kept beside the records, which are not pruned, so this
bounds the table that grows by a row a request without changing any
figure a page shows. At least a day when set.
## Band
### `name`
`string`
### `priority`
`int`
### `saturation_ceiling`
`float64`
## Capabilities
### `tools`
@@ -408,152 +420,6 @@ figure a page shows. At least a day when set.
`bool`
## Config
### `node_id`
`string`
NodeID defaults to the hostname. It must equal the common name of the
agent's certificate, so set it only where the two differ.
### `server`
`string`
Server is the control plane, and Servers is the same thing when there is
more than one. Both are accepted: every configuration written before
there could be a second control plane names one address, and none of
them should have to change.
The order is tried in order, and a refusal moves to the next. Only the
leader accepts an agent, so a standby is skipped in the time it takes to
be told so — which is what makes failover a reconnect rather than an
edit to every node's configuration.
### `servers`
`[]string`
### `hostname`
`string`
Hostname defaults to the machine's.
### `mgmt_addr`
`string`
MgmtAddr is the address engines are reached on. It defaults to this
machine's address on the route to the control plane, which is the one
the control plane and the router can reach.
### `labels`
`map[string]string`
Labels are free-form facts about a node for a profile's node_selector.
Nothing needs them: memory, GPUs, engines and disk are measured, and a
label restating one of those goes stale the day the machine changes.
### `mgmt_interface`
`string`
MgmtInterface is the ethernet interface NCCL bootstraps over. It
defaults to the interface holding MgmtAddr, then eth0.
### `fabrics`
`[]inventory.FabricAttachment`
Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
default subnet prefix is the same on every fabric, so nothing can be
inferred from the wire (plan §5.2).
### `heartbeat_interval`
`time.Duration`
### `reconnect_base`
`time.Duration`
ReconnectBase is the first reconnect delay; it backs off from there.
### `reconnect_max`
`time.Duration`
### `adopt_ready_timeout`
`time.Duration`
AdoptReadyTimeout bounds how long an adopted engine is given to answer a
health check before the agent stops waiting on it.
Shorter than a startup timeout on purpose: this engine is already
running, so it is not loading a model, it is either serving or it is not.
### `tls`
`tlsconf.Config`
TLS authenticates both ends of the control-plane stream. Leaving it unset
keeps the plaintext behaviour, which only belongs on a trusted network.
Takes a [Config](#config) block.
### `model_dirs`
`[]string`
ModelDirs are the directories this node keeps model weights in, beyond
WeightsCache, which is always looked in. Storage differs at every installation,
so llmgw observes it rather than owning it: a Lustre mount, an image
exported over the fabric, local NVMe and a copy made by hand all look
identical from here, and placement only needs to know whether the file
is present.
### `engines_dir`
`string`
EnginesDir is where the agent installs engines, one directory per
build. Defaults to /opt/llmgw/engines.
### `weights_cache`
`string`
WeightsCache is where a remote source is downloaded to. Defaults to the
first entry of ModelDirs, because a node that says where its models live
has already answered this question, and otherwise to
/var/lib/llmgw-weights.
Fetching is the simple arrangement: every node keeps its own copy. The
exotic ones need no code — weights on a shared filer or an image exported
over the fabric are simply present, and present means nothing is fetched.
### `discovery`
`discovery.Options`
Takes a [Options](#options) block.
### `engines`
`engines.Config`
Takes a [Config](#config) block.
### `supervisor`
`supervisor.Config`
Takes a [Config](#config) block.
## Control
### `interval`
@@ -782,7 +648,7 @@ agents use. Empty leaves the connection in plain HTTP, which is only
acceptable on a network where nothing untrusted can reach that port —
the mirror says as much in its own log when it starts that way.
Takes a [Config](#config) block.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `registry`
@@ -1044,16 +910,6 @@ before affinity is abandoned.
`float64`
## Price
### `prompt_per_1k`
`int64`
### `completion_per_1k`
`int64`
## Profile
### `name`
@@ -1425,7 +1281,7 @@ AgentTLS authenticates the fleet-facing port. It deliberately does not
cover the client-facing one: that surface is fronted by a publicly
trusted certificate, while this one is mutual and private.
Takes a [Config](#config) block.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `monitoring`
@@ -1617,3 +1473,345 @@ node and has not been checked against a real cluster.
RayBin is used when LaunchMode is "ray".
## accounting.Price
### `prompt_per_1k`
`int64`
### `completion_per_1k`
`int64`
## admission.Band
### `name`
`string`
### `priority`
`int`
Priority orders bands; higher is served first.
### `saturation_ceiling`
`float64`
SaturationCeiling is the pool saturation above which this band waits.
1.0 means "never hold this band back".
## admission.Config
### `bands`
`[]Band`
### `default_band`
`string`
### `queue_ttl`
`time.Duration`
QueueTTL drops a request that has waited this long. Sending the GPU work
the client has already given up on helps nobody.
### `queue_capacity_per_band`
`int`
QueueCapacityPerBand bounds each band's queue. Rejecting immediately
beats accumulating silently.
### `queue_capacity_total`
`int`
QueueCapacityTotal bounds everything queued, across bands. When it is
reached, a request is not refused while a lower band has requests
queued: the newest of the lowest band is turned away instead
(displacement), since the oldest have waited longest and are nearest
their TTL. Zero takes half of every band's capacity together.
### `interval`
`time.Duration`
Interval is how often the dispatcher re-checks saturation. Releases also
wake it, so this is only the backstop for saturation falling on its own.
## agent.Config
### `node_id`
`string`
NodeID defaults to the hostname. It must equal the common name of the
agent's certificate, so set it only where the two differ.
### `server`
`string`
Server is the control plane, and Servers is the same thing when there is
more than one. Both are accepted: every configuration written before
there could be a second control plane names one address, and none of
them should have to change.
The order is tried in order, and a refusal moves to the next. Only the
leader accepts an agent, so a standby is skipped in the time it takes to
be told so — which is what makes failover a reconnect rather than an
edit to every node's configuration.
### `servers`
`[]string`
### `hostname`
`string`
Hostname defaults to the machine's.
### `mgmt_addr`
`string`
MgmtAddr is the address engines are reached on. It defaults to this
machine's address on the route to the control plane, which is the one
the control plane and the router can reach.
### `labels`
`map[string]string`
Labels are free-form facts about a node for a profile's node_selector.
Nothing needs them: memory, GPUs, engines and disk are measured, and a
label restating one of those goes stale the day the machine changes.
### `mgmt_interface`
`string`
MgmtInterface is the ethernet interface NCCL bootstraps over. It
defaults to the interface holding MgmtAddr, then eth0.
### `fabrics`
`[]inventory.FabricAttachment`
Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
default subnet prefix is the same on every fabric, so nothing can be
inferred from the wire (plan §5.2).
### `heartbeat_interval`
`time.Duration`
### `reconnect_base`
`time.Duration`
ReconnectBase is the first reconnect delay; it backs off from there.
### `reconnect_max`
`time.Duration`
### `adopt_ready_timeout`
`time.Duration`
AdoptReadyTimeout bounds how long an adopted engine is given to answer a
health check before the agent stops waiting on it.
Shorter than a startup timeout on purpose: this engine is already
running, so it is not loading a model, it is either serving or it is not.
### `tls`
`tlsconf.Config`
TLS authenticates both ends of the control-plane stream. Leaving it unset
keeps the plaintext behaviour, which only belongs on a trusted network.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `model_dirs`
`[]string`
ModelDirs are the directories this node keeps model weights in, beyond
WeightsCache, which is always looked in. Storage differs at every installation,
so llmgw observes it rather than owning it: a Lustre mount, an image
exported over the fabric, local NVMe and a copy made by hand all look
identical from here, and placement only needs to know whether the file
is present.
### `engines_dir`
`string`
EnginesDir is where the agent installs engines, one directory per
build. Defaults to /opt/llmgw/engines.
### `weights_cache`
`string`
WeightsCache is where a remote source is downloaded to. Defaults to the
first entry of ModelDirs, because a node that says where its models live
has already answered this question, and otherwise to
/var/lib/llmgw-weights.
Fetching is the simple arrangement: every node keeps its own copy. The
exotic ones need no code — weights on a shared filer or an image exported
over the fabric are simply present, and present means nothing is fetched.
### `discovery`
`discovery.Options`
Takes a [Options](#options) block.
### `engines`
`engines.Config`
Takes a [engines.Config](#enginesconfig) block.
### `supervisor`
`supervisor.Config`
Takes a [supervisor.Config](#supervisorconfig) block.
## config.Band
### `name`
`string`
### `priority`
`int`
### `saturation_ceiling`
`float64`
## config.Price
### `prompt_per_1k`
`int64`
### `completion_per_1k`
`int64`
## engines.Config
### `disabled`
`[]string`
Disabled are engines this node never installs or advertises, whatever
the control plane asks: vLLM on a node of cards no build of it runs on,
that is meant for llama.cpp alone.
### `vllm`
`VLLMConfig`
Takes a [VLLMConfig](#vllmconfig) block.
### `sglang`
`SGLangConfig`
Takes a [SGLangConfig](#sglangconfig) block.
### `llamacpp`
`LlamaCppConfig`
Takes a [LlamaCppConfig](#llamacppconfig) block.
### `sim`
`SimConfig`
Takes a [SimConfig](#simconfig) block.
## supervisor.Config
### `work_dir`
`string`
WorkDir holds pid files and per-shard scratch space.
### `log_lines`
`int`
LogLines is how many lines of engine output to retain per shard.
### `stop_timeout`
`time.Duration`
StopTimeout is how long a process gets to exit after SIGTERM before it is
killed. Engines need time to free GPU memory cleanly.
### `health_interval`
`time.Duration`
HealthInterval is how often a starting shard is polled.
### `rendezvous_timeout`
`time.Duration`
RendezvousTimeout bounds the wait for a multi-node rendezvous point to
start listening.
### `alive_grace`
`time.Duration`
AliveGrace is how long a non-serving rank must stay up before it counts
as ready. There is no external signal for a worker, so staying alive is
the only evidence available.
## tlsconf.Config
### `ca`
`string`
CA is the bundle used to verify the other side. In a real deployment this
is whichever authority issued the fleet's certificates.
### `cert`
`string`
Cert and Key are this component's own identity.
### `key`
`string`
### `insecure`
`bool`
Insecure disables TLS entirely. It exists for local runs against
sim-engines; anything using it must stay on a trusted network.