Every control plane lists the others alike to Prometheus

140 named 150 a store member, 150 named itself a control plane: Prometheus
asking both scraped 150 twice and counted its every series twice, and asked
of 140 alone while 150 led it lost every node. Each control plane now lists
all of them as control planes before the store's members, so two answers
agree and a target is scraped once. The shipped prometheus.yml says to
scrape and ask every control plane.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-15 07:53:22 +02:00
1 parent c22528c26e
commit fc60481e40
4 files changed
+57 -1

No files matched your search

+13 -1
View File
@@ -21,11 +21,18 @@ scrape_configs:
# The control plane's dedicated metrics port. Not its admin listener: that one
# is loopback because it also serves pprof, and pprof can dump the heap where
# API keys sit in plaintext.
#
# Every control plane, one line each: the leader moves on restarts and the
# router's metrics are the leading one's, so scraping one alone leaves gaps
# — and alerts on them — whenever another leads.
- job_name: llmgw
static_configs:
- targets: ['10.0.0.140:9091']
labels:
instance: control-plane
instance: control-plane-140
# - targets: ['10.0.0.150:9091']
# labels:
# instance: control-plane-150
# Every machine of the cluster, for its node exporter: network and disk
# traffic, filesystems, CPU time. The control plane lists them — nodes,
@@ -33,10 +40,15 @@ scrape_configs:
# nothing edited here, and each carries its llmgw id as the node label.
# `llmgw node add` installs the exporter; on a machine added before that,
# apt-get install prometheus-node-exporter.
# Asked of every control plane too: a standby lists only the control
# planes and the store, and they label every machine alike, so the same
# target from two of them is scraped once.
- job_name: nodes
http_sd_configs:
- url: http://10.0.0.140:9091/prometheus/targets
refresh_interval: 1m
# - url: http://10.0.0.150:9091/prometheus/targets
# refresh_interval: 1m
- job_name: prometheus
static_configs: