Setup asks for webhooks, but one left empty was never mentioned again: a lost node or a judge down was told nobody. With no notify.webhooks in the cluster's settings, ctl status and the console's Overview say so, and how to set one. BACKLOG: the judge's refusal after a placement change, tried again on idle VMs, did not happen; a cold judge's first answer is 2-3x slower but inside its budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2c32a58a06
commit
a00c92648e
10 files changed
+60
-4
No files matched your search
+8
-3
@@ -463,8 +463,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
|
||||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||||
templates. Left: `server init` and the console do not yet ask for a
|
||||
receiver — the templates are filled in by hand. Events are claimed in
|
||||
templates. `server init` and both setup screens ask for webhooks and test
|
||||
each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
|
||||
Overview say the alerts go nowhere (2026-11-12). Events are claimed in
|
||||
the store before they are sent (2026-10-10), so two control planes send
|
||||
each once.
|
||||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||||
@@ -2056,7 +2057,11 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
|
||||
node (the controller is its node), so a site wanting control planes without nodes chooses the
|
||||
controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
|
||||
shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
|
||||
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4.
|
||||
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
|
||||
VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
|
||||
first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
|
||||
refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
|
||||
before it takes requests.
|
||||
- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
|
||||
mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
|
||||
(/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's
|
||||
|
||||
Reference in new issue
Block a user