Ilker ManapandClaude Opus 5.5
2a16c26575
Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:
sudo llmgw node add ubuntu@10.0.0.21
The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.
The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.
Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.
llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
63 lines
2.1 KiB
Go
63 lines
2.1 KiB
Go
package onboard
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"os/exec"
|
|
"runtime"
|
|
)
|
|
|
|
// Remote runs a shell command on the machine being onboarded, with input.
|
|
type Remote interface {
|
|
Run(ctx context.Context, script string, stdin []byte) ([]byte, error)
|
|
}
|
|
|
|
// SSH is Remote through the system's ssh, so the operator's keys, agent and
|
|
// ~/.ssh/config apply exactly as they do when they log in by hand.
|
|
type SSH struct {
|
|
Addr, User string
|
|
}
|
|
|
|
// sshOptions never prompt: a question nobody is there to answer would hang
|
|
// onboarding instead of failing it.
|
|
//
|
|
// accept-new rather than refusing unknown host keys: a machine provisioned a
|
|
// minute ago has a key nobody has seen, and the risk guarded against is a
|
|
// substituted machine, whose changed key is still refused.
|
|
var sshOptions = []string{
|
|
"-o", "BatchMode=yes",
|
|
"-o", "ConnectTimeout=10",
|
|
"-o", "StrictHostKeyChecking=accept-new",
|
|
"-o", "ServerAliveInterval=15",
|
|
// "Permanently added … to the list of known hosts" and the like are
|
|
// for a person at a prompt; here they would be read as the answer.
|
|
"-o", "LogLevel=ERROR",
|
|
}
|
|
|
|
// Run runs script with sh on the machine. Files travel on standard input,
|
|
// not scp, which a hardened sshd may not offer.
|
|
func (s *SSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
|
|
target := s.Addr
|
|
if s.User != "" {
|
|
target = s.User + "@" + s.Addr
|
|
}
|
|
args := append(append([]string{}, sshOptions...), target, "sh", "-c", shellQuote(script))
|
|
cmd := exec.CommandContext(ctx, "ssh", args...)
|
|
if stdin != nil {
|
|
cmd.Stdin = bytes.NewReader(stdin)
|
|
}
|
|
// The answer is standard output alone: ssh and the machine's login
|
|
// scripts write to standard error, and mixed in they were taken for the
|
|
// answer — the first connection's "Warning: Permanently added" read as
|
|
// the machine's hostname. Standard error comes back only with a failure,
|
|
// where it is usually the reason.
|
|
var stdout, stderr bytes.Buffer
|
|
cmd.Stdout, cmd.Stderr = &stdout, &stderr
|
|
if err := cmd.Run(); err != nil {
|
|
return append(stdout.Bytes(), stderr.Bytes()...), err
|
|
}
|
|
return stdout.Bytes(), nil
|
|
}
|
|
|
|
func hostArch() string { return runtime.GOARCH }
|