Ilker ManapandClaude Opus 5.5
20deb22dfa
Every agent file on the cluster named its own node, its own address, the path to llama-server and two weight directories — different on every node, kept true by hand, and not true: node05 carried "ram: 32g" beside a measured 31 GB, looked for weights in an empty directory while fetching them into another, and the other four never looked in theirs at all, so four nodes reported no disk. Growing a node's RAM would have gone unnoticed, because memory was read once, at registration. The agent now finds these out. Its id is the hostname, which is also the certificate's common name. Its address is the one on the route to the control plane, which the kernel picks for a UDP dial without sending anything. llama-server is looked for on PATH and then where release archives are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather than a failed discovery — which also cut discovery short before InfiniBand on every CPU node not told to skip GPUs. Fetched weights land in /var/lib/llmgw-weights by default, and wherever they land is always looked in. Memory is read on every heartbeat, and installed engines are probed again each minute and sent when they change, so a VM given more RAM or a node given vLLM becomes a candidate without anyone restarting its agent. The control plane logs both changes, since "the node now qualifies" is the answer to why a model placed an hour after it was asked for. What is written still wins. The agent file shrinks to the control plane's addresses and the TLS files, the same on every node; onboarding writes only the overrides it was given. What cannot be found stays configuration: which fabric an adapter is on, because OpenSM's default prefix is the same on all of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
65 lines
2.0 KiB
Go
65 lines
2.0 KiB
Go
package config
|
|
|
|
import (
|
|
"os"
|
|
"path/filepath"
|
|
"testing"
|
|
)
|
|
|
|
// TestTheDocumentedAgentExampleLoads parses the file shipped in deploy/examples.
|
|
//
|
|
// It exists because the example was wrong for the entire life of the project
|
|
// and nothing noticed. The fabric struct carried no koanf tags, so the `name:`
|
|
// key the example documents bound to nothing, and an agent configured exactly
|
|
// as instructed refused to start with "fabrics[0]: name is required" — naming
|
|
// the key the operator had just written.
|
|
//
|
|
// Every fabric until then had been configured in a Go test, where the tags do
|
|
// not matter. A published example that has never been parsed is a guess.
|
|
func TestTheDocumentedAgentExampleLoads(t *testing.T) {
|
|
path := filepath.Join("..", "..", "deploy", "examples", "agent.yaml")
|
|
if _, err := os.Stat(path); err != nil {
|
|
t.Skipf("no example to check: %v", err)
|
|
}
|
|
|
|
cfg, err := LoadAgent(path)
|
|
if err != nil {
|
|
t.Fatalf("the documented example does not load: %v", err)
|
|
}
|
|
if len(cfg.Fabrics) == 0 {
|
|
t.Fatal("the example documents a fabric and none was parsed")
|
|
}
|
|
f := cfg.Fabrics[0]
|
|
if f.Fabric == "" {
|
|
t.Error("the fabric name did not bind")
|
|
}
|
|
if len(f.HCAs) == 0 {
|
|
t.Error("the fabric's adapters did not bind")
|
|
}
|
|
if f.RateGbps == 0 {
|
|
t.Error("the fabric's rate did not bind")
|
|
}
|
|
// node_id is left to the hostname, so the example need not say it; where
|
|
// the control plane is, nothing can find out.
|
|
if cfg.Server == "" {
|
|
t.Errorf("server=%q", cfg.Server)
|
|
}
|
|
}
|
|
|
|
func TestTheDocumentedServerExampleLoads(t *testing.T) {
|
|
path := filepath.Join("..", "..", "deploy", "examples", "server.yaml")
|
|
if _, err := os.Stat(path); err != nil {
|
|
t.Skipf("no example to check: %v", err)
|
|
}
|
|
cfg, err := Load(path)
|
|
if err != nil {
|
|
t.Fatalf("the documented example does not load: %v", err)
|
|
}
|
|
if err := cfg.Validate(); err != nil {
|
|
t.Fatalf("the documented example does not validate: %v", err)
|
|
}
|
|
if cfg.Listen == "" || len(cfg.Models) == 0 {
|
|
t.Errorf("listen=%q models=%d", cfg.Listen, len(cfg.Models))
|
|
}
|
|
}
|