Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates
notify.webhooks sends signed JSON or Slack-compatible messages, queued, retried, de-duplicated hourly; ctl notify test checks each webhook. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
646ff996c7
commit
1b71c38f89
24 files changed
+907
-4
No files matched your search
@@ -1,10 +1,11 @@
|
||||
# Alertmanager for the llmgw cluster.
|
||||
#
|
||||
# Nothing is sent anywhere yet, on purpose. Where an alert should go — an
|
||||
# Nothing is sent anywhere until a receiver is chosen, on purpose. Where an alert should go — an
|
||||
# address, a Slack workspace, a webhook — is a decision with credentials
|
||||
# attached, and a monitoring system that quietly starts mailing people is worse
|
||||
# than one that is visibly not wired up. Alerts group and fire here and are
|
||||
# visible at :9093 and in Prometheus; add a receiver below to make them travel.
|
||||
# visible at :9093 and in Prometheus; uncomment a receiver template below and
|
||||
# name it on the route to make them travel.
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
|
||||
@@ -31,6 +32,53 @@ route:
|
||||
receivers:
|
||||
- name: none
|
||||
|
||||
# Receiver templates. Uncomment one, fill in where it goes, and name it in
|
||||
# `receiver:` above — on the route, the critical branch, or both. Secrets go
|
||||
# in files Alertmanager reads (the *_file settings), mode 0640 owned by
|
||||
# root:prometheus, not in this file. `amtool check-config` before a restart.
|
||||
#
|
||||
# Slack (an incoming webhook):
|
||||
# - name: slack
|
||||
# slack_configs:
|
||||
# - api_url_file: /etc/prometheus/slack-webhook-url
|
||||
# channel: '#llmgw'
|
||||
# send_resolved: true
|
||||
# title: '{{ .Status | toUpper }}: {{ .CommonLabels.alertname }}'
|
||||
# text: >-
|
||||
# {{ range .Alerts }}{{ .Annotations.summary }}
|
||||
# {{ .Annotations.description }}
|
||||
# {{ end }}
|
||||
#
|
||||
# E-mail (through a relay that takes authenticated submission):
|
||||
# - name: email
|
||||
# email_configs:
|
||||
# - to: ops@example.com
|
||||
# from: llmgw-alerts@example.com
|
||||
# smarthost: smtp.example.com:587
|
||||
# auth_username: llmgw-alerts@example.com
|
||||
# auth_password_file: /etc/prometheus/smtp-password
|
||||
# send_resolved: true
|
||||
#
|
||||
# PagerDuty (Events API v2), for the critical branch — what should wake
|
||||
# someone:
|
||||
# - name: pagerduty
|
||||
# pagerduty_configs:
|
||||
# - routing_key_file: /etc/prometheus/pagerduty-routing-key
|
||||
# severity: '{{ .CommonLabels.severity }}'
|
||||
# description: '{{ .CommonAnnotations.summary }}'
|
||||
#
|
||||
# A webhook of your own (Alertmanager's JSON, version 4):
|
||||
# - name: webhook
|
||||
# webhook_configs:
|
||||
# - url_file: /etc/prometheus/alert-webhook-url
|
||||
# send_resolved: true
|
||||
# max_alerts: 20
|
||||
#
|
||||
# The gateway itself also sends a few events straight to webhooks, with no
|
||||
# Prometheus in between — a model parked, a node lost, a budget spent, new
|
||||
# weights published: `notify.webhooks` in server.yaml (docs/install.md,
|
||||
# "Being told").
|
||||
|
||||
inhibit_rules:
|
||||
# A node that is down explains every replica on it. Reporting both says the
|
||||
# same thing twice and buries whichever is the actual cause.
|
||||
|
||||
Reference in new issue
Block a user