Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates

notify.webhooks sends signed JSON or Slack-compatible messages, queued,
retried, de-duplicated hourly; ctl notify test checks each webhook.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 16:58:11 +02:00
1 parent 646ff996c7
commit 1b71c38f89
24 files changed
+907 -4

No files matched your search

+35
View File
@@ -927,6 +927,41 @@ fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.
### Being told
Prometheus alerts travel once `alertmanager.yml` names a receiver; it ships
templates for Slack, e-mail, PagerDuty and a webhook of your own, commented
out until you choose.
A few things are worth a message even without a monitoring stack, and the
control plane sends those itself:
| Event | When |
|---|---|
| `model_parked` | A model failed every start in a row and is no longer retried (`ctl retry` tries again). |
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
| `update_available` | A model's registry repository published different weights. Once per revision. |
```yaml
notify:
webhooks:
- url: https://hooks.slack.com/services/… # Slack, Mattermost, Rocket.Chat
format: slack
events: [model_parked, node_lost]
- url: https://ops.example.com/llmgw # your own receiver
secret_file: /etc/llmgw/notify-secret # signs the body
```
A `json` webhook (the default) gets the event as JSON — `kind`, `at`,
`subject`, `message`, `detail` — with `X-Llmgw-Event` naming the kind and,
when `secret_file` is set, `X-Llmgw-Signature: sha256=<hex>`, the HMAC-SHA256
of the body under the file's contents; check it before believing the body. A
webhook that does not answer is tried three times and then given up on, and the
same event about the same thing is sent at most once an hour, so a budget that
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
test event to each webhook and says how each answered.
## PostgreSQL, when SQLite is no longer the right shape
SQLite is the default and stays the right answer for most installations: one