Merge ha: replicated store, second control plane, join codes, setup and console screens
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
36cdbce125
67 files changed
+8574
-490
No files matched your search
@@ -430,3 +430,50 @@ groups:
|
||||
What their people feel. Compare with SlowFirstToken per model: if
|
||||
only this tenant is slow, it is the models it uses or its queue
|
||||
band, not the cluster.
|
||||
# The store, when it is a replicated cluster under Patroni (llmgw server
|
||||
# init --patroni). Published only for a store of several hosts.
|
||||
- name: llmgw-store
|
||||
rules:
|
||||
- alert: StoreHasNoLeader
|
||||
expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "The store's cluster has no primary"
|
||||
description: >-
|
||||
Every write fails — keys, usage, budgets, leadership. Patroni
|
||||
promotes a standby within its 30-second TTL; a minute without one
|
||||
means it cannot: etcd has lost its majority, or no standby is fit.
|
||||
`patronictl -c /etc/patroni/config.yml list` on a member says which.
|
||||
|
||||
- alert: StorePatroniUnreachable
|
||||
expr: llmgw_store_patroni_up == 0
|
||||
for: 2m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "No member of the store's cluster answers on its Patroni port"
|
||||
description: >-
|
||||
The control plane reads the cluster's state from port 8008 on each
|
||||
member. The database may still be serving; nothing here can tell.
|
||||
|
||||
- alert: StoreReplicaNotStreaming
|
||||
expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
|
||||
description: >-
|
||||
A standby that is not streaming is not a standby: if the primary
|
||||
is lost now, there is nothing to promote. "missing" is a host of
|
||||
the store address the cluster no longer lists — usually a machine
|
||||
that is down.
|
||||
|
||||
- alert: StoreReplicaBehind
|
||||
expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
|
||||
description: >-
|
||||
Patroni will not promote a standby more than 1 MB behind, so a
|
||||
failover now would find none fit. -1 is a lag Patroni cannot read.
|
||||
Reference in new issue
Block a user