新增主库落在 pve1(HDD)时的回切提示告警
自动切换后主库安全落到 pve1 不会触发无主/多主规则;2026-09-30 的切换一天无人察觉。 Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
@@ -24,6 +24,15 @@ spec:
|
||||
labels: {severity: warning, service: shared-postgresql}
|
||||
annotations:
|
||||
summary: '共享 PG {{ $labels.env }} {{ $labels.member }} 无法以监控角色执行 SQL'
|
||||
# 正常 primary 在 laptop ZFS SSD;pve1 的数据盘是 HDD。自动切换后主库安全落到 pve1
|
||||
# 不会触发任何“无主/多主”规则(2026-09-30 的切换整整一天无人察觉),所以单独提示回切。
|
||||
# 15m 留给计划内切换演练。
|
||||
- alert: SharedPostgresPrimaryOnStandbyHost
|
||||
expr: pg_replication_is_replica{job="shared-postgresql-sql", env="prod", member="pg-pve1"} == 0
|
||||
for: 15m
|
||||
labels: {severity: warning, service: shared-postgresql}
|
||||
annotations:
|
||||
summary: 共享 PG 生产主库运行在 pg-pve1(HDD),请确认切换原因并计划回切 laptop
|
||||
- alert: SharedPostgresSqlCollectorFailing
|
||||
expr: pg_scrape_collector_success{job="shared-postgresql-sql"} == 0 or pg_exporter_last_scrape_error{job="shared-postgresql-sql"} == 1
|
||||
for: 10m
|
||||
|
||||
@@ -137,3 +137,29 @@ tests:
|
||||
- exp_labels: {job: shared-postgresql-sql, instance: 'p:9187', member: pg-pve1, env: prod, severity: warning, service: shared-postgresql}
|
||||
exp_annotations:
|
||||
summary: 共享 PG prod pg-pve1 有事务空闲未提交超过 10 分钟
|
||||
- name: 主库落在 pve1 持续 15 分钟提示回切
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: pg_replication_is_replica{job="shared-postgresql-sql",instance="10.60.0.20:9187",cluster="homelab-pg-prod",member="pg-pve1",env="prod"}
|
||||
values: '0x20'
|
||||
- series: pg_replication_is_replica{job="shared-postgresql-sql",instance="192.168.10.127:9187",cluster="homelab-pg-prod",member="pg-laptop",env="prod"}
|
||||
values: '1x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: SharedPostgresPrimaryOnStandbyHost
|
||||
exp_alerts: []
|
||||
- eval_time: 16m
|
||||
alertname: SharedPostgresPrimaryOnStandbyHost
|
||||
exp_alerts:
|
||||
- exp_labels: {job: shared-postgresql-sql, instance: '10.60.0.20:9187', cluster: homelab-pg-prod, member: pg-pve1, env: prod, severity: warning, service: shared-postgresql}
|
||||
exp_annotations:
|
||||
summary: 共享 PG 生产主库运行在 pg-pve1(HDD),请确认切换原因并计划回切 laptop
|
||||
- name: 主库在 laptop 不提示
|
||||
interval: 1m
|
||||
input_series:
|
||||
- series: pg_replication_is_replica{job="shared-postgresql-sql",instance="10.60.0.20:9187",member="pg-pve1",env="prod"}
|
||||
values: '1x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 16m
|
||||
alertname: SharedPostgresPrimaryOnStandbyHost
|
||||
exp_alerts: []
|
||||
|
||||
Reference in New Issue
Block a user