Status says when the cluster's alerts go nowhere
CI / test (push) Failing after 6m57s

Setup asks for webhooks, but one left empty was never mentioned again: a
lost node or a judge down was told nobody. With no notify.webhooks in the
cluster's settings, ctl status and the console's Overview say so, and how
to set one. BACKLOG: the judge's refusal after a placement change, tried
again on idle VMs, did not happen; a cold judge's first answer is 2-3x
slower but inside its budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-12 22:49:37 +01:00
1 parent 2c32a58a06
commit a00c92648e
10 files changed
+60 -4

No files matched your search

+8 -3
View File
@@ -463,8 +463,9 @@ portal is not done — the page shows; the console and ctl change.
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
lost, budget spent, update available; signed JSON or Slack; retries,
hourly de-duplication), `ctl notify test`, Alertmanager receiver
templates. Left: `server init` and the console do not yet ask for a
receiver — the templates are filled in by hand. Events are claimed in
templates. `server init` and both setup screens ask for webhooks and test
each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
Overview say the alerts go nowhere (2026-11-12). Events are claimed in
the store before they are sent (2026-10-10), so two control planes send
each once.
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
@@ -2056,7 +2057,11 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
node (the controller is its node), so a site wanting control planes without nodes chooses the
controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4.
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
before it takes requests.
- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
(/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's