diff --git a/deploy/monitoring/README.md b/deploy/monitoring/README.md new file mode 100644 index 0000000..efc6802 --- /dev/null +++ b/deploy/monitoring/README.md @@ -0,0 +1,125 @@ +# Watching a cluster + +Prometheus, Alertmanager and Grafana, configured for what llmgw actually +publishes. Every alert here is written against a series this system emits; +nothing is aimed at a metric that might exist. + +## What it is + +One scrape target for the whole fleet. + +Agents publish no metrics of their own. The control plane gathers what they +report — node state, memory, disk, engine queue depth, KV occupancy — and +republishes all of it, so a single endpoint describes every machine. That is +worth knowing before going looking for an exporter on each node. + +## The port it scrapes + +`metrics_listen`, not `admin_listen`. + +The admin listener is bound to loopback for one reason: it also serves pprof, +and pprof can dump the process heap, where API keys sit in plaintext. Metrics +are the one thing on that listener something else genuinely needs to reach, so +they get a port of their own instead of becoming an argument for opening the +other one. + +```yaml +# /etc/llmgw/server.yaml +metrics_listen: ":9091" +``` + +Confirm it carries nothing else: + +``` +curl -s -o /dev/null -w '%{http_code}\n' http://:9091/debug/pprof/ # 404 +curl -s -o /dev/null -w '%{http_code}\n' http://:9091/api/v1/nodes # 404 +``` + +## Installing + +On Ubuntu 24.04 the first two come from the archive; Grafana needs its own +repository. + +``` +apt-get install prometheus prometheus-alertmanager + +install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml +install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml +install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml +promtool check config /etc/prometheus/prometheus.yml +systemctl restart prometheus prometheus-alertmanager + +install -m 0644 grafana/datasource.yaml /etc/grafana/provisioning/datasources/llmgw.yaml +install -m 0644 grafana/dashboards.yaml /etc/grafana/provisioning/dashboards/llmgw.yaml +install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \ + /var/lib/grafana/dashboards/llmgw.json +systemctl restart grafana-server +``` + +The dashboard is provisioned from a file rather than clicked together, so it +survives a reinstall and can be reviewed in a diff. + +## Where to run it + +Not on the control plane. Monitoring that dies with the thing it monitors tells +you nothing at the moment you need it, and the control plane is also the busiest +machine in a small cluster. + +Anywhere else on the management network will do. The mirror is a reasonable +host: it has the disk, and serving weights is bursty rather than constant. + +## Nothing is sent anywhere + +Alertmanager groups and holds alerts and has no receiver. + +Where an alert should go — an address, a Slack workspace, a webhook — is a +decision with credentials attached, and a monitoring system that quietly starts +mailing people is worse than one that is visibly not wired up. Alerts are +visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to +make them travel. `critical` already routes separately so it can be pointed +somewhere else without touching the rest. + +## Check that it fires + +An alerting system nobody has seen fire is an assumption. + +``` +llmgw ctl cordon # work already there keeps running +# ~2 minutes later +curl -s http://:9093/api/v2/alerts | jq '.[].labels' +llmgw ctl uncordon # the alert resolves on its own +``` + +Cordoning is the cheapest real test: it changes a metric the rules read, stops +nothing, and reverses in one command. + +## What the alerts assume + +**A stopped model publishes no series.** `llmgw_model_replicas_ready` exists +only for models with a replica placed, so `== 0` cannot fire for one that is +deliberately off. That is what makes an on-demand catalogue quiet here. + +**Saturation of 1 is the healthy limit, not failure.** It is the worse of queue +depth and KV occupancy, measured by `ctl bench` rather than guessed. The alert +allows five minutes above it before saying anything. + +**Latencies are in seconds.** These thresholds are written for CPU inference. A +p99 first token of thirty seconds is the alerting point here and would be +absurd on a GPU; change it when the hardware changes. + +**Disk is only reported by nodes that hold weights.** A node with no model +directory reports nothing rather than zero, and the rule divides free by total +so a node that reports neither never fires. + +## What is not here + +No exporter on each node, so no host CPU, load average or network. The control +plane publishes memory, disk and liveness for every node, which is what +placement decides on; the rest is a `node_exporter` away if somebody wants it. + +No probe of the mirror. Its `/v1/health` is JSON rather than Prometheus text, +and checking it properly means `blackbox_exporter`. + +No recording rules. At this size every query in the dashboard runs fast enough +raw, and a recording rule is a second definition of a number to keep in step +with the first. diff --git a/deploy/monitoring/alertmanager.yml b/deploy/monitoring/alertmanager.yml new file mode 100644 index 0000000..c4ecb02 --- /dev/null +++ b/deploy/monitoring/alertmanager.yml @@ -0,0 +1,41 @@ +# Alertmanager for the llmgw cluster. +# +# Nothing is sent anywhere yet, on purpose. Where an alert should go — an +# address, a Slack workspace, a webhook — is a decision with credentials +# attached, and a monitoring system that quietly starts mailing people is worse +# than one that is visibly not wired up. Alerts group and fire here and are +# visible at :9093 and in Prometheus; add a receiver below to make them travel. +global: + resolve_timeout: 5m + +route: + # By alert and by what it is about, so five nodes going quiet in the same + # network blip arrive as one notification rather than five. + group_by: ['alertname', 'node', 'model'] + group_wait: 30s + group_interval: 5m + # Long, because a repeat is a reminder and not news. A cluster with one thing + # wrong should not produce a message an hour. + repeat_interval: 12h + receiver: none + + routes: + # Critical gets its own branch so it can be pointed somewhere else later + # without touching the rest: the difference between "a node is draining" + # and "nothing is serving" is who should be woken. + - matchers: [severity="critical"] + receiver: none + group_wait: 10s + repeat_interval: 4h + +receivers: + - name: none + +inhibit_rules: + # A node that is down explains every replica on it. Reporting both says the + # same thing twice and buries whichever is the actual cause. + - source_matchers: [alertname="NodeNotReady"] + target_matchers: [alertname="EndpointMetricsStale"] + equal: ['node'] + - source_matchers: [alertname="ControlPlaneUnreachable"] + target_matchers: [severity="warning"] diff --git a/deploy/monitoring/grafana/dashboards.yaml b/deploy/monitoring/grafana/dashboards.yaml new file mode 100644 index 0000000..76c3e81 --- /dev/null +++ b/deploy/monitoring/grafana/dashboards.yaml @@ -0,0 +1,11 @@ +apiVersion: 1 +providers: + - name: llmgw + orgId: 1 + folder: llmgw + type: file + disableDeletion: false + updateIntervalSeconds: 30 + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards diff --git a/deploy/monitoring/grafana/datasource.yaml b/deploy/monitoring/grafana/datasource.yaml new file mode 100644 index 0000000..80f5a77 --- /dev/null +++ b/deploy/monitoring/grafana/datasource.yaml @@ -0,0 +1,11 @@ +# Provisioned rather than clicked in, so the dashboard below can reference it by +# a fixed uid and the whole setup survives a reinstall of the machine. +apiVersion: 1 +datasources: + - name: Prometheus + uid: llmgw-prom + type: prometheus + access: proxy + url: http://localhost:9090 + isDefault: true + editable: false diff --git a/deploy/monitoring/grafana/llmgw-overview.json b/deploy/monitoring/grafana/llmgw-overview.json new file mode 100644 index 0000000..5b36716 --- /dev/null +++ b/deploy/monitoring/grafana/llmgw-overview.json @@ -0,0 +1,492 @@ +{ + "uid": "llmgw-overview", + "title": "llmgw \u2014 cluster", + "tags": [ + "llmgw" + ], + "timezone": "browser", + "schemaVersion": 39, + "version": 1, + "refresh": "30s", + "time": { + "from": "now-6h", + "to": "now" + }, + "panels": [ + { + "id": 1, + "title": "Nodes ready", + "type": "stat", + "description": "Nodes taking work. A node that is known but draining or cordoned is not counted.", + "gridPos": { + "x": 0, + "y": 0, + "w": 4, + "h": 4 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "count(llmgw_node_up == 1)", + "legendFormat": "ready", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + }, + "textMode": "auto", + "colorMode": "value" + } + }, + { + "id": 2, + "title": "Replicas serving", + "type": "stat", + "description": "Across every model. A model deliberately stopped contributes nothing and is not a fault.", + "gridPos": { + "x": 4, + "y": 0, + "w": 4, + "h": 4 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "sum(llmgw_model_replicas_ready)", + "legendFormat": "ready", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + }, + "textMode": "auto", + "colorMode": "value" + } + }, + { + "id": 3, + "title": "Models with something serving", + "type": "stat", + "description": "", + "gridPos": { + "x": 8, + "y": 0, + "w": 4, + "h": 4 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "count(llmgw_model_replicas_ready > 0)", + "legendFormat": "models", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + }, + "textMode": "auto", + "colorMode": "value" + } + }, + { + "id": 4, + "title": "Requests per second", + "type": "timeseries", + "description": "Failing is counted separately rather than as a percentage, so a quiet cluster reads as quiet instead of as 0% of nothing.", + "gridPos": { + "x": 12, + "y": 0, + "w": 12, + "h": 4 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "sum(rate(llmgw_requests_total[5m]))", + "legendFormat": "all", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + }, + { + "refId": "B", + "expr": "sum(rate(llmgw_requests_total{status=~\"5..\"}[5m]))", + "legendFormat": "failing", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "reqps" + }, + "overrides": [] + } + }, + { + "id": 5, + "title": "Replicas ready, by model", + "type": "timeseries", + "description": "Only models with a replica placed appear. One that is stopped on purpose publishes no series.", + "gridPos": { + "x": 0, + "y": 4, + "w": 12, + "h": 7 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_model_replicas_ready", + "legendFormat": "{{model}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + } + }, + { + "id": 6, + "title": "Pool saturation", + "type": "timeseries", + "description": "The worse of queue depth and KV occupancy. 1 is the limit measured as healthy, not the point of failure.", + "gridPos": { + "x": 12, + "y": 4, + "w": 12, + "h": 7 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_pool_saturation", + "legendFormat": "{{model}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "max": 2, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + }, + "overrides": [] + } + }, + { + "id": 7, + "title": "Time to first token", + "type": "timeseries", + "description": "As the router sees it. On CPU these are seconds, not milliseconds; compare against what ctl bench measured.", + "gridPos": { + "x": 0, + "y": 11, + "w": 12, + "h": 7 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "histogram_quantile(0.5, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))", + "legendFormat": "{{model}} p50", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + }, + { + "refId": "B", + "expr": "histogram_quantile(0.99, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[5m])))", + "legendFormat": "{{model}} p99", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + } + }, + { + "id": 8, + "title": "Tokens per second", + "type": "timeseries", + "description": "", + "gridPos": { + "x": 12, + "y": 11, + "w": 12, + "h": 7 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "sum by (direction) (rate(llmgw_tokens_total[5m]))", + "legendFormat": "{{direction}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + } + }, + { + "id": 9, + "title": "Engine queue depth", + "type": "timeseries", + "description": "Requests the engine has accepted and not started. Rising here is the first sign of saturation.", + "gridPos": { + "x": 0, + "y": 18, + "w": 8, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_endpoint_waiting", + "legendFormat": "{{replica}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + } + }, + { + "id": 10, + "title": "KV cache occupancy", + "type": "timeseries", + "description": "", + "gridPos": { + "x": 8, + "y": 18, + "w": 8, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_endpoint_kv_cache_usage", + "legendFormat": "{{replica}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "percentunit" + }, + "overrides": [] + } + }, + { + "id": 11, + "title": "Model disk free", + "type": "timeseries", + "description": "The filesystems holding weights. A node that cannot hold the next model is skipped by placement rather than failing.", + "gridPos": { + "x": 16, + "y": 18, + "w": 8, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_node_disk_free_bytes", + "legendFormat": "{{node}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes" + }, + "overrides": [] + } + }, + { + "id": 12, + "title": "Seconds since each node last reported", + "type": "timeseries", + "description": "Heartbeats are every 5s. A line that climbs is an agent going quiet; the control plane stops placing there at 15s.", + "gridPos": { + "x": 0, + "y": 24, + "w": 12, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "llmgw_node_last_seen_seconds", + "legendFormat": "{{node}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + }, + "overrides": [] + } + }, + { + "id": 13, + "title": "Replicas by state", + "type": "timeseries", + "description": "", + "gridPos": { + "x": 12, + "y": 24, + "w": 12, + "h": 6 + }, + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + }, + "targets": [ + { + "refId": "A", + "expr": "sum by (state) (llmgw_replicas)", + "legendFormat": "{{state}}", + "datasource": { + "type": "prometheus", + "uid": "llmgw-prom" + } + } + ], + "fieldConfig": { + "defaults": {}, + "overrides": [] + } + } + ] +} \ No newline at end of file diff --git a/deploy/monitoring/llmgw-rules.yml b/deploy/monitoring/llmgw-rules.yml new file mode 100644 index 0000000..520efa0 --- /dev/null +++ b/deploy/monitoring/llmgw-rules.yml @@ -0,0 +1,130 @@ +# Alerts for llmgw. Every expression here is against a series this cluster +# actually publishes; nothing is written against a metric that might exist. +groups: + - name: llmgw-fleet + rules: + - alert: ControlPlaneUnreachable + expr: up{job="llmgw"} == 0 + for: 1m + labels: { severity: critical } + annotations: + summary: "The control plane is not answering" + description: >- + Nothing has scraped 10.0.0.140:9091 for a minute. Engines already + running keep serving — the router is in the same process, so if + this is down, so is the gateway. + + - alert: NodeNotReady + expr: llmgw_node_up == 0 + for: 2m + labels: { severity: warning } + annotations: + summary: "{{ $labels.node }} is {{ $labels.state }}" + description: >- + The node is known but not taking work. Two minutes, because a node + restarting its agent is briefly not ready and is not a problem. + + - alert: AgentGoneQuiet + expr: llmgw_node_last_seen_seconds > 60 + for: 1m + labels: { severity: warning } + annotations: + summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}" + description: >- + Heartbeats are every 5s. A minute of silence is the network or the + agent, and the control plane stages expiry at 15s unreachable and + 60s lost — so this fires around the point it stops placing there. + + - alert: ModelHasNothingServing + expr: llmgw_model_replicas_ready == 0 + for: 10m + labels: { severity: critical } + annotations: + summary: "{{ $labels.model }} has no replica ready" + description: >- + The model is placed and nothing is serving it. Ten minutes, because + a large model on CPU legitimately takes minutes to load. A model + that is deliberately stopped publishes no series at all, so this + cannot fire for one nobody asked for. + + - alert: PoolSaturated + expr: llmgw_pool_saturation > 1 + for: 5m + labels: { severity: warning } + annotations: + summary: "{{ $labels.model }} is over its healthy limit" + description: >- + Saturation is the worse of queue depth and KV occupancy; 1 is the + limit measured as healthy, not the point of failure. Sustained above + it means requests are queueing — add a replica or lower the load. + + - alert: EndpointBreakerOpen + expr: llmgw_endpoint_breaker_open == 1 + for: 2m + labels: { severity: warning } + annotations: + summary: "{{ $labels.replica }} has been taken out of rotation" + description: >- + The router stopped sending to this replica after repeated failures. + Traffic is going elsewhere; if it is the only replica, the model is + down. + + - alert: EndpointMetricsStale + expr: llmgw_endpoint_metrics_stale == 1 + for: 5m + labels: { severity: warning } + annotations: + summary: "{{ $labels.replica }} is being routed to blind" + description: >- + Its queue depth and KV usage are old, so the scheduler is choosing + with stale numbers. The replica may be serving perfectly well; the + decisions about it are the problem. + + - alert: NodeDiskFilling + expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10 + for: 15m + labels: { severity: warning } + annotations: + summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free" + description: >- + Weights are tens of gigabytes. A node that cannot hold the next one + is skipped by placement rather than failing, so this is quiet until + somebody wonders why a model will not land there. + + - name: llmgw-traffic + rules: + # These need traffic through the gateway to have any series at all. A + # counter with no observations publishes nothing, so a quiet cluster + # simply has no data here rather than a rule that is always false. + - alert: HighErrorRate + expr: >- + sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m])) + / sum by (model) (rate(llmgw_requests_total[5m])) > 0.05 + for: 5m + labels: { severity: critical } + annotations: + summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests" + + - alert: SlowFirstToken + expr: >- + histogram_quantile(0.99, + sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30 + for: 10m + labels: { severity: warning } + annotations: + summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99" + description: >- + On CPU this is expected under load and is why the threshold is + thirty seconds rather than one. Compare with what ctl bench measured + before treating it as a fault. + + - alert: RequestsBeingRejected + expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0 + for: 5m + labels: { severity: warning } + annotations: + summary: "Admission is turning away {{ $labels.band }} traffic" + description: >- + Working as designed when the cluster is full: the lowest band is + held first. Worth knowing about because the callers being refused + do not know why. diff --git a/deploy/monitoring/prometheus.yml b/deploy/monitoring/prometheus.yml index 0af2a81..bcddc97 100644 --- a/deploy/monitoring/prometheus.yml +++ b/deploy/monitoring/prometheus.yml @@ -1,45 +1,32 @@ -# Scraping llmgw. +# Prometheus for the llmgw cluster. # -# Point this at metrics_listen, not admin_listen. They are different ports for -# a reason: the admin listener also carries pprof, which can dump the process -# heap, and API keys are in that heap in plaintext. -# -# In the control plane's configuration: -# -# admin_listen: "127.0.0.1:9090" # dashboard, management API, pprof -# metrics_listen: "0.0.0.0:9091" # metrics only, safe to reach -# -# One target for the whole fleet. Node and GPU figures are exported by the -# control plane, which already holds them — agents report hardware at -# registration and state on every heartbeat — so there is nothing to scrape on -# the nodes and no Prometheus configuration per machine. +# One scrape target for the whole fleet. Agents publish nothing of their own — +# the control plane gathers what they report and republishes it, so node health, +# disk and endpoint state all arrive here from 10.0.0.140. global: scrape_interval: 15s + evaluation_interval: 15s + external_labels: + cluster: mynodes + +alerting: + alertmanagers: + - static_configs: + - targets: ['localhost:9093'] + +rule_files: + - /etc/prometheus/rules/llmgw.yml scrape_configs: + # The control plane's dedicated metrics port. Not its admin listener: that one + # is loopback because it also serves pprof, and pprof can dump the heap where + # API keys sit in plaintext. - job_name: llmgw static_configs: - - targets: ["llmgw-ctl.internal:9091"] + - targets: ['10.0.0.140:9091'] labels: - cluster: primary + instance: control-plane -# Worth alerting on, in rough order of how often it matters. -# -# A node that has stopped reporting: -# llmgw_node_last_seen_seconds > 60 -# -# A model that should be serving and is not. Covers a replica that will never -# start as well as one that has died, which look the same from a distance: -# llmgw_model_replicas_ready == 0 -# -# A card kept out of placement by a fault, which is otherwise invisible until -# a model will not fit: -# llmgw_gpu_healthy == 0 -# -# A fleet holding models nobody is asking for. Memory used says a model is -# loaded; utilisation says whether it is doing anything: -# avg_over_time(llmgw_gpu_utilization_percent[1h]) < 5 -# -# Requests queueing because the cluster is full, which is the pair of figures -# that needed two systems to correlate before these were on one endpoint: -# llmgw_admission_queue_depth > 0 + - job_name: prometheus + static_configs: + - targets: ['localhost:9090']