From b921fa0d6bece1aa07965d1e0a8fdf4a165cb563 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 27 Sep 2026 18:06:47 +0200 Subject: [PATCH] Watch the cluster, and prove the alerts fire MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prometheus, Alertmanager and Grafana, with every rule written against a series this system actually publishes. Nothing here is aimed at a metric that might exist. One scrape target for the whole fleet. Agents publish nothing of their own — the control plane gathers what they report and republishes it — which is worth writing down before somebody goes looking for an exporter to install on each node. It scrapes metrics_listen, not the admin listener. That one is loopback because it also serves pprof, and pprof can dump the heap where API keys sit in plaintext. Metrics are the one thing on it something else genuinely needs, so they get a port of their own rather than becoming a reason to open the other. The README shows how to confirm the new port carries nothing else; it answers 404 to both pprof and the admin API. Nothing is sent anywhere. Alertmanager groups and holds and has no receiver: where an alert should go is a decision with credentials attached, and a monitoring system that quietly starts mailing people is worse than one that is visibly not wired up. critical already routes separately so it can be pointed somewhere else without touching the rest. The rules encode what the metrics mean rather than round numbers. A saturation of 1 is the healthy limit measured by ctl bench, not the point of failure, so the alert waits five minutes above it. A thirty-second p99 first token is the threshold because this is CPU inference and would be absurd on a GPU. A model that is deliberately stopped publishes no series at all, so "nothing serving" cannot fire for one nobody asked for — which is what lets an on-demand catalogue sit quiet. Inhibitions, because a node that is down explains every replica on it and reporting both buries the cause. The dashboard is provisioned from a file rather than clicked together, so it survives a reinstall and can be read in a diff. Checked by firing one: cordoning a node moved llmgw_node_up to 0, the rule went pending and then firing after its two minutes, the alert arrived at Alertmanager with the node in its labels, and uncordoning resolved it. An alerting system nobody has seen fire is an assumption. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/monitoring/README.md | 125 +++++ deploy/monitoring/alertmanager.yml | 41 ++ deploy/monitoring/grafana/dashboards.yaml | 11 + deploy/monitoring/grafana/datasource.yaml | 11 + deploy/monitoring/grafana/llmgw-overview.json | 492 ++++++++++++++++++ deploy/monitoring/llmgw-rules.yml | 130 +++++ deploy/monitoring/prometheus.yml | 59 +-- 7 files changed, 833 insertions(+), 36 deletions(-) create mode 100644 deploy/monitoring/README.md create mode 100644 deploy/monitoring/alertmanager.yml create mode 100644 deploy/monitoring/grafana/dashboards.yaml create mode 100644 deploy/monitoring/grafana/datasource.yaml create mode 100644 deploy/monitoring/grafana/llmgw-overview.json create mode 100644 deploy/monitoring/llmgw-rules.yml diff --git a/deploy/monitoring/README.md b/deploy/monitoring/README.md new file mode 100644 index 0000000..efc6802 --- /dev/null +++ b/deploy/monitoring/README.md @@ -0,0 +1,125 @@ +# Watching a cluster + +Prometheus, Alertmanager and Grafana, configured for what llmgw actually +publishes. Every alert here is written against a series this system emits; +nothing is aimed at a metric that might exist. + +## What it is + +One scrape target for the whole fleet. + +Agents publish no metrics of their own. The control plane gathers what they +report — node state, memory, disk, engine queue depth, KV occupancy — and +republishes all of it, so a single endpoint describes every machine. That is +worth knowing before going looking for an exporter on each node. + +## The port it scrapes + +`metrics_listen`, not `admin_listen`. + +The admin listener is bound to loopback for one reason: it also serves pprof, +and pprof can dump the process heap, where API keys sit in plaintext. Metrics +are the one thing on that listener something else genuinely needs to reach, so +they get a port of their own instead of becoming an argument for opening the +other one. + +```yaml +# /etc/llmgw/server.yaml +metrics_listen: ":9091" +``` + +Confirm it carries nothing else: + +``` +curl -s -o /dev/null -w '%{http_code}\n' http://:9091/debug/pprof/ # 404 +curl -s -o /dev/null -w '%{http_code}\n' http://:9091/api/v1/nodes # 404 +``` + +## Installing + +On Ubuntu 24.04 the first two come from the archive; Grafana needs its own +repository. + +``` +apt-get install prometheus prometheus-alertmanager + +install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml +install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml +install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml +promtool check config /etc/prometheus/prometheus.yml +systemctl restart prometheus prometheus-alertmanager + +install -m 0644 grafana/datasource.yaml /etc/grafana/provisioning/datasources/llmgw.yaml +install -m 0644 grafana/dashboards.yaml /etc/grafana/provisioning/dashboards/llmgw.yaml +install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \ + /var/lib/grafana/dashboards/llmgw.json +systemctl restart grafana-server +``` + +The dashboard is provisioned from a file rather than clicked together, so it +survives a reinstall and can be reviewed in a diff. + +## Where to run it + +Not on the control plane. Monitoring that dies with the thing it monitors tells +you nothing at the moment you need it, and the control plane is also the busiest +machine in a small cluster. + +Anywhere else on the management network will do. The mirror is a reasonable +host: it has the disk, and serving weights is bursty rather than constant. + +## Nothing is sent anywhere + +Alertmanager groups and holds alerts and has no receiver. + +Where an alert should go — an address, a Slack workspace, a webhook — is a +decision with credentials attached, and a monitoring system that quietly starts +mailing people is worse than one that is visibly not wired up. Alerts are +visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to +make them travel. `critical` already routes separately so it can be pointed +somewhere else without touching the rest. + +## Check that it fires + +An alerting system nobody has seen fire is an assumption. + +``` +llmgw ctl cordon # work already there keeps running +# ~2 minutes later +curl -s http://:9093/api/v2/alerts | jq '.[].labels' +llmgw ctl uncordon # the alert resolves on its own +``` + +Cordoning is the cheapest real test: it changes a metric the rules read, stops +nothing, and reverses in one command. + +## What the alerts assume + +**A stopped model publishes no series.** `llmgw_model_replicas_ready` exists +only for models with a replica placed, so `== 0` cannot fire for one that is +deliberately off. That is what makes an on-demand catalogue quiet here. + +**Saturation of 1 is the healthy limit, not failure.** It is the worse of queue +depth and KV occupancy, measured by `ctl bench` rather than guessed. The alert +allows five minutes above it before saying anything. + +**Latencies are in seconds.** These thresholds are written for CPU inference. A +p99 first token of thirty seconds is the alerting point here and would be +absurd on a GPU; change it when the hardware changes. + +**Disk is only reported by nodes that hold weights.** A node with no model +directory reports nothing rather than zero, and the rule divides free by total +so a node that reports neither never fires. + +## What is not here + +No exporter on each node, so no host CPU, load average or network. The control +plane publishes memory, disk and liveness for every node, which is what +placement decides on; the rest is a `node_exporter` away if somebody wants it. + +No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text, +and checking it properly means `blackbox_exporter`. + +No recording rules. At this size every query in the dashboard runs fast enough +raw, and a recording rule is a second definition of a number to keep in step +with the first. diff --git a/deploy/monitoring/alertmanager.yml b/deploy/monitoring/alertmanager.yml new file mode 100644 index 0000000..c4ecb02 --- /dev/null +++ b/deploy/monitoring/alertmanager.yml @@ -0,0 +1,41 @@ +# Alertmanager for the llmgw cluster. +# +# Nothing is sent anywhere yet, on purpose. Where an alert should go — an +# address, a Slack workspace, a webhook — is a decision with credentials +# attached, and a monitoring system that quietly starts mailing people is worse +# than one that is visibly not wired up. Alerts group and fire here and are +# visible at :9093 and in Prometheus; add a receiver below to make them travel. +global: + resolve_timeout: 5m + +route: + # By alert and by what it is about, so five nodes going quiet in the same + # network blip arrive as one notification rather than five. + group_by: ['alertname', 'node', 'model'] + group_wait: 30s + group_interval: 5m + # Long, because a repeat is a reminder and not news. A cluster with one thing + # wrong should not produce a message an hour. + repeat_interval: 12h + receiver: none + + routes: + # Critical gets its own branch so it can be pointed somewhere else later + # without touching the rest: the difference between "a node is draining" + # and "nothing is serving" is who should be woken. + - matchers: [severity="critical"] + receiver: none + group_wait: 10s + repeat_interval: 4h + +receivers: + - name: none + +inhibit_rules: + # A node that is down explains every replica on it. Reporting both says the + # same thing twice and buries whichever is the actual cause. + - source_matchers: [alertname="NodeNotReady"] + target_matchers: [alertname="EndpointMetricsStale"] + equal: ['node'] + - source_matchers: [alertname="ControlPlaneUnreachable"] + target_matchers: [severity="warning"] diff --git a/deploy/monitoring/grafana/dashboards.yaml b/deploy/monitoring/grafana/dashboards.yaml new file mode 100644 index 0000000..76c3e81 --- /dev/null +++ b/deploy/monitoring/grafana/dashboards.yaml @@ -0,0 +1,11 @@ +apiVersion: 1 +providers: + - name: llmgw + orgId: 1 + folder: llmgw + type: file + disableDeletion: false + updateIntervalSeconds: 30 + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards diff --git a/deploy/monitoring/grafana/datasource.yaml b/deploy/monitoring/grafana/datasource.yaml new file mode 100644 index 0000000..80f5a77 --- /dev/null +++ b/deploy/monitoring/grafana/datasource.yaml @@ -0,0 +1,11 @@ +# Provisioned rather than clicked in, so the dashboard below can reference it by +# a fixed uid and the whole setup survives a reinstall of the machine. +apiVersion: 1 +datasources: + - name: Prometheus + uid: llmgw-prom + type: prometheus + access: proxy + url: http://localhost:9090 + isDefault: true + editable: false diff --git a/deploy/monitoring/grafana/llmgw-overview.json b/deploy/monitoring/grafana/llmgw-overview.json new file mode 100644 index 0000000..5b36716 --- /dev/null +++ b/deploy/monitoring/grafana/llmgw-overview.json @@ -0,0 +1,492 @@ +{ + "uid": "llmgw-overview", + "title": "llmgw \u2014 cluster", + "tags": [ + "llmgw" + ], + "timezone": "browser", + "schemaVersion": 39, + "version": 1, + "refresh": "30s", + "time": { + "from": "now-6h", + "to": "now" + }, + "panels": [ + { + "id": 1, + "title": "Nodes ready", + "type": "stat", + "description": "Nodes taking work. A node that is known but draining or cordoned is not counted.", + "gridPos": { + "x": 0, + "y": 0, + "w": 4, + "h": 4 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "count(llmgw_node_up == 1)", + "legendFormat": "ready", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + }, + "textMode": "auto", + "colorMode": "value" + } + }, + { + "id": 2, + "title": "Replicas serving", + "type": "stat", + "description": "Across every model. A model deliberately stopped contributes nothing and is not a fault.", + "gridPos": { + "x": 4, + "y": 0, + "w": 4, + "h": 4 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "sum(llmgw_model_replicas_ready)", + "legendFormat": "ready", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + }, + "textMode": "auto", + "colorMode": "value" + } + }, + { + "id": 3, + "title": "Models with something serving", + "type": "stat", + "description": "", + "gridPos": { + "x": 8, + "y": 0, + "w": 4, + "h": 4 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "count(llmgw_model_replicas_ready > 0)", + "legendFormat": "models", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + }, + "textMode": "auto", + "colorMode": "value" + } + }, + { + "id": 4, + "title": "Requests per second", + "type": "timeseries", + "description": "Failing is counted separately rather than as a percentage, so a quiet cluster reads as quiet instead of as 0% of nothing.", + "gridPos": { + "x": 12, + "y": 0, + "w": 12, + "h": 4 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "sum(rate(llmgw_requests_total[5m]))", + "legendFormat": "all", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + }, + { + "refId": "B", + "expr": "sum(rate(llmgw_requests_total{status=~\"5..\"}[5m]))", + "legendFormat": "failing", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "reqps" + }, + "overrides": [] + } + }, + { + "id": 5, + "title": "Replicas ready, by model", + "type": "timeseries", + "description": "Only models with a replica placed appear. One that is stopped on purpose publishes no series.", + "gridPos": { + "x": 0, + "y": 4, + "w": 12, + "h": 7 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_model_replicas_ready", + "legendFormat": "{{model}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + } + }, + { + "id": 6, + "title": "Pool saturation", + "type": "timeseries", + "description": "The worse of queue depth and KV occupancy. 1 is the limit measured as healthy, not the point of failure.", + "gridPos": { + "x": 12, + "y": 4, + "w": 12, + "h": 7 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_pool_saturation", + "legendFormat": "{{model}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "max": 2, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + }, + "overrides": [] + } + }, + { + "id": 7, + "title": "Time to first token", + "type": "timeseries", + "description": "As the router sees it. On CPU these are seconds, not milliseconds; compare against what ctl bench measured.", + "gridPos": { + "x": 0, + "y": 11, + "w": 12, + "h": 7 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "histogram_quantile(0.5, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))", + "legendFormat": "{{model}} p50", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + }, + { + "refId": "B", + "expr": "histogram_quantile(0.99, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))", + "legendFormat": "{{model}} p99", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + } + }, + { + "id": 8, + "title": "Tokens per second", + "type": "timeseries", + "description": "", + "gridPos": { + "x": 12, + "y": 11, + "w": 12, + "h": 7 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "sum by (direction) (rate(llmgw_tokens_total[5m]))", + "legendFormat": "{{direction}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + } + }, + { + "id": 9, + "title": "Engine queue depth", + "type": "timeseries", + "description": "Requests the engine has accepted and not started. Rising here is the first sign of saturation.", + "gridPos": { + "x": 0, + "y": 18, + "w": 8, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_endpoint_waiting", + "legendFormat": "{{replica}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + } + }, + { + "id": 10, + "title": "KV cache occupancy", + "type": "timeseries", + "description": "", + "gridPos": { + "x": 8, + "y": 18, + "w": 8, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_endpoint_kv_cache_usage", + "legendFormat": "{{replica}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "percentunit" + }, + "overrides": [] + } + }, + { + "id": 11, + "title": "Model disk free", + "type": "timeseries", + "description": "The filesystems holding weights. A node that cannot hold the next model is skipped by placement rather than failing.", + "gridPos": { + "x": 16, + "y": 18, + "w": 8, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_node_disk_free_bytes", + "legendFormat": "{{node}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes" + }, + "overrides": [] + } + }, + { + "id": 12, + "title": "Seconds since each node last reported", + "type": "timeseries", + "description": "Heartbeats are every 5s. A line that climbs is an agent going quiet; the control plane stops placing there at 15s.", + "gridPos": { + "x": 0, + "y": 24, + "w": 12, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_node_last_seen_seconds", + "legendFormat": "{{node}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + } + }, + { + "id": 13, + "title": "Replicas by state", + "type": "timeseries", + "description": "", + "gridPos": { + "x": 12, + "y": 24, + "w": 12, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "sum by (state) (llmgw_replicas)", + "legendFormat": "{{state}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + } + } + ] +} \ No newline at end of file diff --git a/deploy/monitoring/llmgw-rules.yml b/deploy/monitoring/llmgw-rules.yml new file mode 100644 index 0000000..520efa0 --- /dev/null +++ b/deploy/monitoring/llmgw-rules.yml @@ -0,0 +1,130 @@ +# Alerts for llmgw. Every expression here is against a series this cluster +# actually publishes; nothing is written against a metric that might exist. +groups: + - name: llmgw-fleet + rules: + - alert: ControlPlaneUnreachable + expr: up{job="llmgw"} == 0 + for: 1m + labels: { severity: critical } + annotations: + summary: "The control plane is not answering" + description: >- + Nothing has scraped 10.0.0.140:9091 for a minute. Engines already + running keep serving — the router is in the same process, so if + this is down, so is the gateway. + + - alert: NodeNotReady + expr: llmgw_node_up == 0 + for: 2m + labels: { severity: warning } + annotations: + summary: "{{ $labels.node }} is {{ $labels.state }}" + description: >- + The node is known but not taking work. Two minutes, because a node + restarting its agent is briefly not ready and is not a problem. + + - alert: AgentGoneQuiet + expr: llmgw_node_last_seen_seconds > 60 + for: 1m + labels: { severity: warning } + annotations: + summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}" + description: >- + Heartbeats are every 5s. A minute of silence is the network or the + agent, and the control plane stages expiry at 15s unreachable and + 60s lost — so this fires around the point it stops placing there. + + - alert: ModelHasNothingServing + expr: llmgw_model_replicas_ready == 0 + for: 10m + labels: { severity: critical } + annotations: + summary: "{{ $labels.model }} has no replica ready" + description: >- + The model is placed and nothing is serving it. Ten minutes, because + a large model on CPU legitimately takes minutes to load. A model + that is deliberately stopped publishes no series at all, so this + cannot fire for one nobody asked for. + + - alert: PoolSaturated + expr: llmgw_pool_saturation > 1 + for: 5m + labels: { severity: warning } + annotations: + summary: "{{ $labels.model }} is over its healthy limit" + description: >- + Saturation is the worse of queue depth and KV occupancy; 1 is the + limit measured as healthy, not the point of failure. Sustained above + it means requests are queueing — add a replica or lower the load. + + - alert: EndpointBreakerOpen + expr: llmgw_endpoint_breaker_open == 1 + for: 2m + labels: { severity: warning } + annotations: + summary: "{{ $labels.replica }} has been taken out of rotation" + description: >- + The router stopped sending to this replica after repeated failures. + Traffic is going elsewhere; if it is the only replica, the model is + down. + + - alert: EndpointMetricsStale + expr: llmgw_endpoint_metrics_stale == 1 + for: 5m + labels: { severity: warning } + annotations: + summary: "{{ $labels.replica }} is being routed to blind" + description: >- + Its queue depth and KV usage are old, so the scheduler is choosing + with stale numbers. The replica may be serving perfectly well; the + decisions about it are the problem. + + - alert: NodeDiskFilling + expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10 + for: 15m + labels: { severity: warning } + annotations: + summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free" + description: >- + Weights are tens of gigabytes. A node that cannot hold the next one + is skipped by placement rather than failing, so this is quiet until + somebody wonders why a model will not land there. + + - name: llmgw-traffic + rules: + # These need traffic through the gateway to have any series at all. A + # counter with no observations publishes nothing, so a quiet cluster + # simply has no data here rather than a rule that is always false. + - alert: HighErrorRate + expr: >- + sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m])) + / sum by (model) (rate(llmgw_requests_total[5m])) > 0.05 + for: 5m + labels: { severity: critical } + annotations: + summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests" + + - alert: SlowFirstToken + expr: >- + histogram_quantile(0.99, + sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30 + for: 10m + labels: { severity: warning } + annotations: + summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99" + description: >- + On CPU this is expected under load and is why the threshold is + thirty seconds rather than one. Compare with what ctl bench measured + before treating it as a fault. + + - alert: RequestsBeingRejected + expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0 + for: 5m + labels: { severity: warning } + annotations: + summary: "Admission is turning away {{ $labels.band }} traffic" + description: >- + Working as designed when the cluster is full: the lowest band is + held first. Worth knowing about because the callers being refused + do not know why. diff --git a/deploy/monitoring/prometheus.yml b/deploy/monitoring/prometheus.yml index 0af2a81..bcddc97 100644 --- a/deploy/monitoring/prometheus.yml +++ b/deploy/monitoring/prometheus.yml @@ -1,45 +1,32 @@ -# Scraping llmgw. +# Prometheus for the llmgw cluster. # -# Point this at metrics_listen, not admin_listen. They are different ports for -# a reason: the admin listener also carries pprof, which can dump the process -# heap, and API keys are in that heap in plaintext. -# -# In the control plane's configuration: -# -# admin_listen: "127.0.0.1:9090" # dashboard, management API, pprof -# metrics_listen: "0.0.0.0:9091" # metrics only, safe to reach -# -# One target for the whole fleet. Node and GPU figures are exported by the -# control plane, which already holds them — agents report hardware at -# registration and state on every heartbeat — so there is nothing to scrape on -# the nodes and no Prometheus configuration per machine. +# One scrape target for the whole fleet. Agents publish nothing of their own — +# the control plane gathers what they report and republishes it, so node health, +# disk and endpoint state all arrive here from 10.0.0.140. global: scrape_interval: 15s + evaluation_interval: 15s + external_labels: + cluster: mynodes + +alerting: + alertmanagers: + - static_configs: + - targets: ['localhost:9093'] + +rule_files: + - /etc/prometheus/rules/llmgw.yml scrape_configs: + # The control plane's dedicated metrics port. Not its admin listener: that one + # is loopback because it also serves pprof, and pprof can dump the heap where + # API keys sit in plaintext. - job_name: llmgw static_configs: - - targets: ["llmgw-ctl.internal:9091"] + - targets: ['10.0.0.140:9091'] labels: - cluster: primary + instance: control-plane -# Worth alerting on, in rough order of how often it matters. -# -# A node that has stopped reporting: -# llmgw_node_last_seen_seconds > 60 -# -# A model that should be serving and is not. Covers a replica that will never -# start as well as one that has died, which look the same from a distance: -# llmgw_model_replicas_ready == 0 -# -# A card kept out of placement by a fault, which is otherwise invisible until -# a model will not fit: -# llmgw_gpu_healthy == 0 -# -# A fleet holding models nobody is asking for. Memory used says a model is -# loaded; utilisation says whether it is doing anything: -# avg_over_time(llmgw_gpu_utilization_percent[1h]) < 5 -# -# Requests queueing because the cluster is full, which is the pair of figures -# that needed two systems to correlate before these were on one endpoint: -# llmgw_admission_queue_depth > 0 + - job_name: prometheus + static_configs: + - targets: ['localhost:9090']