Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates
notify.webhooks sends signed JSON or Slack-compatible messages, queued, retried, de-duplicated hourly; ctl notify test checks each webhook. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
646ff996c7
commit
1b71c38f89
24 files changed
+907
-4
No files matched your search
@@ -927,6 +927,41 @@ fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
### Being told
|
||||
|
||||
Prometheus alerts travel once `alertmanager.yml` names a receiver; it ships
|
||||
templates for Slack, e-mail, PagerDuty and a webhook of your own, commented
|
||||
out until you choose.
|
||||
|
||||
A few things are worth a message even without a monitoring stack, and the
|
||||
control plane sends those itself:
|
||||
|
||||
| Event | When |
|
||||
|---|---|
|
||||
| `model_parked` | A model failed every start in a row and is no longer retried (`ctl retry` tries again). |
|
||||
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
|
||||
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
|
||||
| `update_available` | A model's registry repository published different weights. Once per revision. |
|
||||
|
||||
```yaml
|
||||
notify:
|
||||
webhooks:
|
||||
- url: https://hooks.slack.com/services/… # Slack, Mattermost, Rocket.Chat
|
||||
format: slack
|
||||
events: [model_parked, node_lost]
|
||||
- url: https://ops.example.com/llmgw # your own receiver
|
||||
secret_file: /etc/llmgw/notify-secret # signs the body
|
||||
```
|
||||
|
||||
A `json` webhook (the default) gets the event as JSON — `kind`, `at`,
|
||||
`subject`, `message`, `detail` — with `X-Llmgw-Event` naming the kind and,
|
||||
when `secret_file` is set, `X-Llmgw-Signature: sha256=<hex>`, the HMAC-SHA256
|
||||
of the body under the file's contents; check it before believing the body. A
|
||||
webhook that does not answer is tried three times and then given up on, and the
|
||||
same event about the same thing is sent at most once an hour, so a budget that
|
||||
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
|
||||
test event to each webhook and says how each answered.
|
||||
|
||||
## PostgreSQL, when SQLite is no longer the right shape
|
||||
|
||||
SQLite is the default and stays the right answer for most installations: one
|
||||
|
||||
Reference in new issue
Block a user