Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates
notify.webhooks sends signed JSON or Slack-compatible messages, queued, retried, de-duplicated hourly; ctl notify test checks each webhook. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
646ff996c7
commit
1b71c38f89
24 files changed
+907
-4
No files matched your search
@@ -927,6 +927,41 @@ fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
### Being told
|
||||
|
||||
Prometheus alerts travel once `alertmanager.yml` names a receiver; it ships
|
||||
templates for Slack, e-mail, PagerDuty and a webhook of your own, commented
|
||||
out until you choose.
|
||||
|
||||
A few things are worth a message even without a monitoring stack, and the
|
||||
control plane sends those itself:
|
||||
|
||||
| Event | When |
|
||||
|---|---|
|
||||
| `model_parked` | A model failed every start in a row and is no longer retried (`ctl retry` tries again). |
|
||||
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
|
||||
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
|
||||
| `update_available` | A model's registry repository published different weights. Once per revision. |
|
||||
|
||||
```yaml
|
||||
notify:
|
||||
webhooks:
|
||||
- url: https://hooks.slack.com/services/… # Slack, Mattermost, Rocket.Chat
|
||||
format: slack
|
||||
events: [model_parked, node_lost]
|
||||
- url: https://ops.example.com/llmgw # your own receiver
|
||||
secret_file: /etc/llmgw/notify-secret # signs the body
|
||||
```
|
||||
|
||||
A `json` webhook (the default) gets the event as JSON — `kind`, `at`,
|
||||
`subject`, `message`, `detail` — with `X-Llmgw-Event` naming the kind and,
|
||||
when `secret_file` is set, `X-Llmgw-Signature: sha256=<hex>`, the HMAC-SHA256
|
||||
of the body under the file's contents; check it before believing the body. A
|
||||
webhook that does not answer is tried three times and then given up on, and the
|
||||
same event about the same thing is sent at most once an hour, so a budget that
|
||||
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
|
||||
test event to each webhook and says how each answered.
|
||||
|
||||
## PostgreSQL, when SQLite is no longer the right shape
|
||||
|
||||
SQLite is the default and stays the right answer for most installations: one
|
||||
|
||||
@@ -230,6 +230,7 @@ rather than repeated.
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `guard` | `guard.Config` | [Server](#server) |
|
||||
| `notify` | `notify.Config` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
@@ -256,6 +257,10 @@ rather than repeated.
|
||||
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `url` | `string` | [Webhook](#webhook) |
|
||||
| `format` | `string` | [Webhook](#webhook) |
|
||||
| `events` | `[]string` | [Webhook](#webhook) |
|
||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
@@ -298,6 +303,8 @@ rather than repeated.
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
| `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) |
|
||||
| `host` | `string` | [notify.Config](#notifyconfig) |
|
||||
| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
|
||||
| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
|
||||
| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
|
||||
@@ -1868,6 +1875,16 @@ guard.default, and to parts of the tenancy tree by attaching them
|
||||
|
||||
Takes a [guard.Config](#guardconfig) block.
|
||||
|
||||
### `notify`
|
||||
|
||||
`notify.Config`
|
||||
|
||||
Notify sends what needs a person — a model parked, a node lost, a
|
||||
budget spent, new weights published — to webhooks, signed, without a
|
||||
Prometheus and Alertmanager in between. Empty sends nothing.
|
||||
|
||||
Takes a [notify.Config](#notifyconfig) block.
|
||||
|
||||
### `firewall_networks`
|
||||
|
||||
`[]string`
|
||||
@@ -2056,6 +2073,33 @@ node and has not been checked against a real cluster.
|
||||
|
||||
RayBin is used when LaunchMode is "ray".
|
||||
|
||||
## Webhook
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
### `format`
|
||||
|
||||
`string`
|
||||
|
||||
Format is json (the event, with X-Llmgw-Signature when SecretFile is
|
||||
set) or slack ({"text": …}, which Slack, Mattermost and Rocket.Chat
|
||||
incoming webhooks take).
|
||||
|
||||
### `events`
|
||||
|
||||
`[]string`
|
||||
|
||||
Events are the kinds sent here; empty is all.
|
||||
|
||||
### `secret_file`
|
||||
|
||||
`string`
|
||||
|
||||
SecretFile holds the key the body is signed with: X-Llmgw-Signature
|
||||
is sha256=<hex HMAC of the body>.
|
||||
|
||||
## accounting.Price
|
||||
|
||||
### `prompt_per_1k`
|
||||
@@ -2340,6 +2384,18 @@ Takes a [SimConfig](#simconfig) block.
|
||||
|
||||
`map[string]Policy`
|
||||
|
||||
## notify.Config
|
||||
|
||||
### `webhooks`
|
||||
|
||||
`[]Webhook`
|
||||
|
||||
### `host`
|
||||
|
||||
`string`
|
||||
|
||||
Host names the gateway in messages; empty is this machine's name.
|
||||
|
||||
## supervisor.Config
|
||||
|
||||
### `work_dir`
|
||||
|
||||
Reference in new issue
Block a user