Add a node with one command: llmgw node add, with a terminal screen
Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:
sudo llmgw node add ubuntu@10.0.0.21
The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.
The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.
Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.
llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
c8da06988f
commit
2a16c26575
17 files changed
+2056
-120
No files matched your search
@@ -74,6 +74,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /api/v1/weights/collect", a.collectWeights)
|
||||
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
|
||||
mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
|
||||
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
|
||||
}
|
||||
|
||||
@@ -812,6 +813,35 @@ func (a *API) Explain(model string) ExplainView {
|
||||
// One endpoint rather than three verbs, because they are the same operation
|
||||
// with different targets and the difference matters: cordon stops new
|
||||
// placement, drain also moves existing work off.
|
||||
// removeNode forgets a node that has gone for good: a machine retired or
|
||||
// rebuilt under another name, which otherwise stays listed as lost until the
|
||||
// control plane restarts.
|
||||
//
|
||||
// Only one that is not there: lost, or unreachable with nothing placed on
|
||||
// it. A connected node removed here would register again on its next
|
||||
// heartbeat, and one with replicas would take them out of the reconciler's
|
||||
// sight while they may still be running.
|
||||
func (a *API) removeNode(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
inv := a.ctl.Inventory()
|
||||
n, ok := inv.Get(id)
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotFound, fmt.Sprintf("no node %q", id))
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case n.State == inventory.Lost:
|
||||
case n.State == inventory.Unreachable && len(inv.Allocations(id)) == 0:
|
||||
default:
|
||||
writeError(w, http.StatusConflict, fmt.Sprintf(
|
||||
"%s is %s: stop its agent (systemctl stop llmgw-agent) and remove it once it is lost, a minute later", id, n.State))
|
||||
return
|
||||
}
|
||||
inv.Remove(id)
|
||||
a.ctl.Reconciler().Notify()
|
||||
writeJSON(w, http.StatusOK, map[string]string{"id": id, "removed": "true"})
|
||||
}
|
||||
|
||||
func (a *API) setNodeState(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
|
||||
|
||||
Reference in new issue
Block a user