Watch the cluster, and prove the alerts fire
CI / test (push) Successful in 1m45s

Prometheus, Alertmanager and Grafana, with every rule written against a series
this system actually publishes. Nothing here is aimed at a metric that might
exist.

One scrape target for the whole fleet. Agents publish nothing of their own —
the control plane gathers what they report and republishes it — which is worth
writing down before somebody goes looking for an exporter to install on each
node.

It scrapes metrics_listen, not the admin listener. That one is loopback because
it also serves pprof, and pprof can dump the heap where API keys sit in
plaintext. Metrics are the one thing on it something else genuinely needs, so
they get a port of their own rather than becoming a reason to open the other.
The README shows how to confirm the new port carries nothing else; it answers
404 to both pprof and the admin API.

Nothing is sent anywhere. Alertmanager groups and holds and has no receiver:
where an alert should go is a decision with credentials attached, and a
monitoring system that quietly starts mailing people is worse than one that is
visibly not wired up. critical already routes separately so it can be pointed
somewhere else without touching the rest.

The rules encode what the metrics mean rather than round numbers. A saturation
of 1 is the healthy limit measured by ctl bench, not the point of failure, so
the alert waits five minutes above it. A thirty-second p99 first token is the
threshold because this is CPU inference and would be absurd on a GPU. A model
that is deliberately stopped publishes no series at all, so "nothing serving"
cannot fire for one nobody asked for — which is what lets an on-demand
catalogue sit quiet.

Inhibitions, because a node that is down explains every replica on it and
reporting both buries the cause.

The dashboard is provisioned from a file rather than clicked together, so it
survives a reinstall and can be read in a diff.

Checked by firing one: cordoning a node moved llmgw_node_up to 0, the rule went
pending and then firing after its two minutes, the alert arrived at Alertmanager
with the node in its labels, and uncordoning resolved it. An alerting system
nobody has seen fire is an assumption.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 18:06:47 +02:00
1 parent 8fd4c1392d
commit b921fa0d6b
7 files changed
+833 -36

No files matched your search

+125
View File
@@ -0,0 +1,125 @@
# Watching a cluster
Prometheus, Alertmanager and Grafana, configured for what llmgw actually
publishes. Every alert here is written against a series this system emits;
nothing is aimed at a metric that might exist.
## What it is
One scrape target for the whole fleet.
Agents publish no metrics of their own. The control plane gathers what they
report — node state, memory, disk, engine queue depth, KV occupancy — and
republishes all of it, so a single endpoint describes every machine. That is
worth knowing before going looking for an exporter on each node.
## The port it scrapes
`metrics_listen`, not `admin_listen`.
The admin listener is bound to loopback for one reason: it also serves pprof,
and pprof can dump the process heap, where API keys sit in plaintext. Metrics
are the one thing on that listener something else genuinely needs to reach, so
they get a port of their own instead of becoming an argument for opening the
other one.
```yaml
# /etc/llmgw/server.yaml
metrics_listen: ":9091"
```
Confirm it carries nothing else:
```
curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/debug/pprof/ # 404
curl -s -o /dev/null -w '%{http_code}\n' http://<control-plane>:9091/api/v1/nodes # 404
```
## Installing
On Ubuntu 24.04 the first two come from the archive; Grafana needs its own
repository.
```
apt-get install prometheus prometheus-alertmanager
install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml
install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml
install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml
promtool check config /etc/prometheus/prometheus.yml
systemctl restart prometheus prometheus-alertmanager
install -m 0644 grafana/datasource.yaml /etc/grafana/provisioning/datasources/llmgw.yaml
install -m 0644 grafana/dashboards.yaml /etc/grafana/provisioning/dashboards/llmgw.yaml
install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
/var/lib/grafana/dashboards/llmgw.json
systemctl restart grafana-server
```
The dashboard is provisioned from a file rather than clicked together, so it
survives a reinstall and can be reviewed in a diff.
## Where to run it
Not on the control plane. Monitoring that dies with the thing it monitors tells
you nothing at the moment you need it, and the control plane is also the busiest
machine in a small cluster.
Anywhere else on the management network will do. The mirror is a reasonable
host: it has the disk, and serving weights is bursty rather than constant.
## Nothing is sent anywhere
Alertmanager groups and holds alerts and has no receiver.
Where an alert should go — an address, a Slack workspace, a webhook — is a
decision with credentials attached, and a monitoring system that quietly starts
mailing people is worse than one that is visibly not wired up. Alerts are
visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
make them travel. `critical` already routes separately so it can be pointed
somewhere else without touching the rest.
## Check that it fires
An alerting system nobody has seen fire is an assumption.
```
llmgw ctl cordon <node> # work already there keeps running
# ~2 minutes later
curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
llmgw ctl uncordon <node> # the alert resolves on its own
```
Cordoning is the cheapest real test: it changes a metric the rules read, stops
nothing, and reverses in one command.
## What the alerts assume
**A stopped model publishes no series.** `llmgw_model_replicas_ready` exists
only for models with a replica placed, so `== 0` cannot fire for one that is
deliberately off. That is what makes an on-demand catalogue quiet here.
**Saturation of 1 is the healthy limit, not failure.** It is the worse of queue
depth and KV occupancy, measured by `ctl bench` rather than guessed. The alert
allows five minutes above it before saying anything.
**Latencies are in seconds.** These thresholds are written for CPU inference. A
p99 first token of thirty seconds is the alerting point here and would be
absurd on a GPU; change it when the hardware changes.
**Disk is only reported by nodes that hold weights.** A node with no model
directory reports nothing rather than zero, and the rule divides free by total
so a node that reports neither never fires.
## What is not here
No exporter on each node, so no host CPU, load average or network. The control
plane publishes memory, disk and liveness for every node, which is what
placement decides on; the rest is a `node_exporter` away if somebody wants it.
No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text,
and checking it properly means `blackbox_exporter`.
No recording rules. At this size every query in the dashboard runs fast enough
raw, and a recording rule is a second definition of a number to keep in step
with the first.
+41
View File
@@ -0,0 +1,41 @@
# Alertmanager for the llmgw cluster.
#
# Nothing is sent anywhere yet, on purpose. Where an alert should go — an
# address, a Slack workspace, a webhook — is a decision with credentials
# attached, and a monitoring system that quietly starts mailing people is worse
# than one that is visibly not wired up. Alerts group and fire here and are
# visible at :9093 and in Prometheus; add a receiver below to make them travel.
global:
resolve_timeout: 5m
route:
# By alert and by what it is about, so five nodes going quiet in the same
# network blip arrive as one notification rather than five.
group_by: ['alertname', 'node', 'model']
group_wait: 30s
group_interval: 5m
# Long, because a repeat is a reminder and not news. A cluster with one thing
# wrong should not produce a message an hour.
repeat_interval: 12h
receiver: none
routes:
# Critical gets its own branch so it can be pointed somewhere else later
# without touching the rest: the difference between "a node is draining"
# and "nothing is serving" is who should be woken.
- matchers: [severity="critical"]
receiver: none
group_wait: 10s
repeat_interval: 4h
receivers:
- name: none
inhibit_rules:
# A node that is down explains every replica on it. Reporting both says the
# same thing twice and buries whichever is the actual cause.
- source_matchers: [alertname="NodeNotReady"]
target_matchers: [alertname="EndpointMetricsStale"]
equal: ['node']
- source_matchers: [alertname="ControlPlaneUnreachable"]
target_matchers: [severity="warning"]
+11
View File
@@ -0,0 +1,11 @@
apiVersion: 1
providers:
- name: llmgw
orgId: 1
folder: llmgw
type: file
disableDeletion: false
updateIntervalSeconds: 30
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
+11
View File
@@ -0,0 +1,11 @@
# Provisioned rather than clicked in, so the dashboard below can reference it by
# a fixed uid and the whole setup survives a reinstall of the machine.
apiVersion: 1
datasources:
- name: Prometheus
uid: llmgw-prom
type: prometheus
access: proxy
url: http://localhost:9090
isDefault: true
editable: false
@@ -0,0 +1,492 @@
{
"uid": "llmgw-overview",
"title": "llmgw \u2014 cluster",
"tags": [
"llmgw"
],
"timezone": "browser",
"schemaVersion": 39,
"version": 1,
"refresh": "30s",
"time": {
"from": "now-6h",
"to": "now"
},
"panels": [
{
"id": 1,
"title": "Nodes ready",
"type": "stat",
"description": "Nodes taking work. A node that is known but draining or cordoned is not counted.",
"gridPos": {
"x": 0,
"y": 0,
"w": 4,
"h": 4
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "count(llmgw_node_up == 1)",
"legendFormat": "ready",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
]
},
"textMode": "auto",
"colorMode": "value"
}
},
{
"id": 2,
"title": "Replicas serving",
"type": "stat",
"description": "Across every model. A model deliberately stopped contributes nothing and is not a fault.",
"gridPos": {
"x": 4,
"y": 0,
"w": 4,
"h": 4
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum(llmgw_model_replicas_ready)",
"legendFormat": "ready",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
]
},
"textMode": "auto",
"colorMode": "value"
}
},
{
"id": 3,
"title": "Models with something serving",
"type": "stat",
"description": "",
"gridPos": {
"x": 8,
"y": 0,
"w": 4,
"h": 4
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "count(llmgw_model_replicas_ready > 0)",
"legendFormat": "models",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
]
},
"textMode": "auto",
"colorMode": "value"
}
},
{
"id": 4,
"title": "Requests per second",
"type": "timeseries",
"description": "Failing is counted separately rather than as a percentage, so a quiet cluster reads as quiet instead of as 0% of nothing.",
"gridPos": {
"x": 12,
"y": 0,
"w": 12,
"h": 4
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(llmgw_requests_total[5m]))",
"legendFormat": "all",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
},
{
"refId": "B",
"expr": "sum(rate(llmgw_requests_total{status=~\"5..\"}[5m]))",
"legendFormat": "failing",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps"
},
"overrides": []
}
},
{
"id": 5,
"title": "Replicas ready, by model",
"type": "timeseries",
"description": "Only models with a replica placed appear. One that is stopped on purpose publishes no series.",
"gridPos": {
"x": 0,
"y": 4,
"w": 12,
"h": 7
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "llmgw_model_replicas_ready",
"legendFormat": "{{model}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {},
"overrides": []
}
},
{
"id": 6,
"title": "Pool saturation",
"type": "timeseries",
"description": "The worse of queue depth and KV occupancy. 1 is the limit measured as healthy, not the point of failure.",
"gridPos": {
"x": 12,
"y": 4,
"w": 12,
"h": 7
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "llmgw_pool_saturation",
"legendFormat": "{{model}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"max": 2,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
}
},
"overrides": []
}
},
{
"id": 7,
"title": "Time to first token",
"type": "timeseries",
"description": "As the router sees it. On CPU these are seconds, not milliseconds; compare against what ctl bench measured.",
"gridPos": {
"x": 0,
"y": 11,
"w": 12,
"h": 7
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))",
"legendFormat": "{{model}} p50",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
},
{
"refId": "B",
"expr": "histogram_quantile(0.99, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))",
"legendFormat": "{{model}} p99",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 8,
"title": "Tokens per second",
"type": "timeseries",
"description": "",
"gridPos": {
"x": 12,
"y": 11,
"w": 12,
"h": 7
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum by (direction) (rate(llmgw_tokens_total[5m]))",
"legendFormat": "{{direction}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 9,
"title": "Engine queue depth",
"type": "timeseries",
"description": "Requests the engine has accepted and not started. Rising here is the first sign of saturation.",
"gridPos": {
"x": 0,
"y": 18,
"w": 8,
"h": 6
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "llmgw_endpoint_waiting",
"legendFormat": "{{replica}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {},
"overrides": []
}
},
{
"id": 10,
"title": "KV cache occupancy",
"type": "timeseries",
"description": "",
"gridPos": {
"x": 8,
"y": 18,
"w": 8,
"h": 6
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "llmgw_endpoint_kv_cache_usage",
"legendFormat": "{{replica}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
}
},
{
"id": 11,
"title": "Model disk free",
"type": "timeseries",
"description": "The filesystems holding weights. A node that cannot hold the next model is skipped by placement rather than failing.",
"gridPos": {
"x": 16,
"y": 18,
"w": 8,
"h": 6
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "llmgw_node_disk_free_bytes",
"legendFormat": "{{node}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes"
},
"overrides": []
}
},
{
"id": 12,
"title": "Seconds since each node last reported",
"type": "timeseries",
"description": "Heartbeats are every 5s. A line that climbs is an agent going quiet; the control plane stops placing there at 15s.",
"gridPos": {
"x": 0,
"y": 24,
"w": 12,
"h": 6
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "llmgw_node_last_seen_seconds",
"legendFormat": "{{node}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 13,
"title": "Replicas by state",
"type": "timeseries",
"description": "",
"gridPos": {
"x": 12,
"y": 24,
"w": 12,
"h": 6
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum by (state) (llmgw_replicas)",
"legendFormat": "{{state}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {},
"overrides": []
}
}
]
}
+130
View File
@@ -0,0 +1,130 @@
# Alerts for llmgw. Every expression here is against a series this cluster
# actually publishes; nothing is written against a metric that might exist.
groups:
- name: llmgw-fleet
rules:
- alert: ControlPlaneUnreachable
expr: up{job="llmgw"} == 0
for: 1m
labels: { severity: critical }
annotations:
summary: "The control plane is not answering"
description: >-
Nothing has scraped 10.0.0.140:9091 for a minute. Engines already
running keep serving — the router is in the same process, so if
this is down, so is the gateway.
- alert: NodeNotReady
expr: llmgw_node_up == 0
for: 2m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} is {{ $labels.state }}"
description: >-
The node is known but not taking work. Two minutes, because a node
restarting its agent is briefly not ready and is not a problem.
- alert: AgentGoneQuiet
expr: llmgw_node_last_seen_seconds > 60
for: 1m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}"
description: >-
Heartbeats are every 5s. A minute of silence is the network or the
agent, and the control plane stages expiry at 15s unreachable and
60s lost — so this fires around the point it stops placing there.
- alert: ModelHasNothingServing
expr: llmgw_model_replicas_ready == 0
for: 10m
labels: { severity: critical }
annotations:
summary: "{{ $labels.model }} has no replica ready"
description: >-
The model is placed and nothing is serving it. Ten minutes, because
a large model on CPU legitimately takes minutes to load. A model
that is deliberately stopped publishes no series at all, so this
cannot fire for one nobody asked for.
- alert: PoolSaturated
expr: llmgw_pool_saturation > 1
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.model }} is over its healthy limit"
description: >-
Saturation is the worse of queue depth and KV occupancy; 1 is the
limit measured as healthy, not the point of failure. Sustained above
it means requests are queueing — add a replica or lower the load.
- alert: EndpointBreakerOpen
expr: llmgw_endpoint_breaker_open == 1
for: 2m
labels: { severity: warning }
annotations:
summary: "{{ $labels.replica }} has been taken out of rotation"
description: >-
The router stopped sending to this replica after repeated failures.
Traffic is going elsewhere; if it is the only replica, the model is
down.
- alert: EndpointMetricsStale
expr: llmgw_endpoint_metrics_stale == 1
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.replica }} is being routed to blind"
description: >-
Its queue depth and KV usage are old, so the scheduler is choosing
with stale numbers. The replica may be serving perfectly well; the
decisions about it are the problem.
- alert: NodeDiskFilling
expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10
for: 15m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free"
description: >-
Weights are tens of gigabytes. A node that cannot hold the next one
is skipped by placement rather than failing, so this is quiet until
somebody wonders why a model will not land there.
- name: llmgw-traffic
rules:
# These need traffic through the gateway to have any series at all. A
# counter with no observations publishes nothing, so a quiet cluster
# simply has no data here rather than a rule that is always false.
- alert: HighErrorRate
expr: >-
sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m]))
/ sum by (model) (rate(llmgw_requests_total[5m])) > 0.05
for: 5m
labels: { severity: critical }
annotations:
summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests"
- alert: SlowFirstToken
expr: >-
histogram_quantile(0.99,
sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99"
description: >-
On CPU this is expected under load and is why the threshold is
thirty seconds rather than one. Compare with what ctl bench measured
before treating it as a fault.
- alert: RequestsBeingRejected
expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0
for: 5m
labels: { severity: warning }
annotations:
summary: "Admission is turning away {{ $labels.band }} traffic"
description: >-
Working as designed when the cluster is full: the lowest band is
held first. Worth knowing about because the callers being refused
do not know why.
+23 -36
View File
@@ -1,45 +1,32 @@
# Scraping llmgw.
# Prometheus for the llmgw cluster.
#
# Point this at metrics_listen, not admin_listen. They are different ports for
# a reason: the admin listener also carries pprof, which can dump the process
# heap, and API keys are in that heap in plaintext.
#
# In the control plane's configuration:
#
# admin_listen: "127.0.0.1:9090" # dashboard, management API, pprof
# metrics_listen: "0.0.0.0:9091" # metrics only, safe to reach
#
# One target for the whole fleet. Node and GPU figures are exported by the
# control plane, which already holds them — agents report hardware at
# registration and state on every heartbeat — so there is nothing to scrape on
# the nodes and no Prometheus configuration per machine.
# One scrape target for the whole fleet. Agents publish nothing of their own —
# the control plane gathers what they report and republishes it, so node health,
# disk and endpoint state all arrive here from 10.0.0.140.
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: mynodes
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
rule_files:
- /etc/prometheus/rules/llmgw.yml
scrape_configs:
# The control plane's dedicated metrics port. Not its admin listener: that one
# is loopback because it also serves pprof, and pprof can dump the heap where
# API keys sit in plaintext.
- job_name: llmgw
static_configs:
- targets: ["llmgw-ctl.internal:9091"]
- targets: ['10.0.0.140:9091']
labels:
cluster: primary
instance: control-plane
# Worth alerting on, in rough order of how often it matters.
#
# A node that has stopped reporting:
# llmgw_node_last_seen_seconds > 60
#
# A model that should be serving and is not. Covers a replica that will never
# start as well as one that has died, which look the same from a distance:
# llmgw_model_replicas_ready == 0
#
# A card kept out of placement by a fault, which is otherwise invisible until
# a model will not fit:
# llmgw_gpu_healthy == 0
#
# A fleet holding models nobody is asking for. Memory used says a model is
# loaded; utilisation says whether it is doing anything:
# avg_over_time(llmgw_gpu_utilization_percent[1h]) < 5
#
# Requests queueing because the cluster is full, which is the pair of figures
# that needed two systems to correlate before these were on one endpoint:
# llmgw_admission_queue_depth > 0
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']