新增主库落在 pve1(HDD)时的回切提示告警
yaml / yaml (pull_request) Successful in 2m25s
ansible / lint (pull_request) Successful in 4m3s
ansible / collection-test (pull_request) Successful in 4m12s

自动切换后主库安全落到 pve1 不会触发无主/多主规则;2026-09-30 的切换一天无人察觉。

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-10-01 16:30:45 +00:00
co-authored by Claude Opus 5.5
parent 954bb39a23
commit 943d31bf08
3 changed files with 36 additions and 1 deletions
+1 -1
View File
@@ -116,7 +116,7 @@ dev 为 laptop `:9188`,仅监听实例内网地址。exporter 以实例 OS 用
规则 `platform/observability/metrics/rules/shared-postgresql-sql.yaml`(PrometheusRule)以上游 规则 `platform/observability/metrics/rules/shared-postgresql-sql.yaml`(PrometheusRule)以上游
`postgres_mixin` 为底改写:采集/SQL 不通、复制回放落后 > 1 MiB(与 failover 阈值一致)、主库复制槽 `postgres_mixin` 为底改写:采集/SQL 不通、复制回放落后 > 1 MiB(与 failover 阈值一致)、主库复制槽
保留 > 1 GiB、WAL 归档失败、XID/MXID 年龄、客户端连接 > 80%、空闲事务 > 10 分钟、事务 > 1 小时、 保留 > 1 GiB、主库落在 pve1(HDD)超过 15 分钟、WAL 归档失败、XID/MXID 年龄、客户端连接 > 80%、空闲事务 > 10 分钟、事务 > 1 小时、
死锁、频繁请求 checkpoint。目标带 `env` 标签,Alertmanager 把 `env="dev"` 送到独立 dev 频道。 死锁、频繁请求 checkpoint。目标带 `env` 标签,Alertmanager 把 `env="dev"` 送到独立 dev 频道。
2026-10-01 验证:隔离集成测试覆盖 peer+ident 登录(主/备/dev)、读取 `pg_stat_statements`、 2026-10-01 验证:隔离集成测试覆盖 peer+ident 登录(主/备/dev)、读取 `pg_stat_statements`、
@@ -24,6 +24,15 @@ spec:
labels: {severity: warning, service: shared-postgresql} labels: {severity: warning, service: shared-postgresql}
annotations: annotations:
summary: '共享 PG {{ $labels.env }} {{ $labels.member }} 无法以监控角色执行 SQL' summary: '共享 PG {{ $labels.env }} {{ $labels.member }} 无法以监控角色执行 SQL'
# 正常 primary 在 laptop ZFS SSD;pve1 的数据盘是 HDD。自动切换后主库安全落到 pve1
# 不会触发任何“无主/多主”规则(2026-09-30 的切换整整一天无人察觉),所以单独提示回切。
# 15m 留给计划内切换演练。
- alert: SharedPostgresPrimaryOnStandbyHost
expr: pg_replication_is_replica{job="shared-postgresql-sql", env="prod", member="pg-pve1"} == 0
for: 15m
labels: {severity: warning, service: shared-postgresql}
annotations:
summary: 共享 PG 生产主库运行在 pg-pve1(HDD),请确认切换原因并计划回切 laptop
- alert: SharedPostgresSqlCollectorFailing - alert: SharedPostgresSqlCollectorFailing
expr: pg_scrape_collector_success{job="shared-postgresql-sql"} == 0 or pg_exporter_last_scrape_error{job="shared-postgresql-sql"} == 1 expr: pg_scrape_collector_success{job="shared-postgresql-sql"} == 0 or pg_exporter_last_scrape_error{job="shared-postgresql-sql"} == 1
for: 10m for: 10m
@@ -137,3 +137,29 @@ tests:
- exp_labels: {job: shared-postgresql-sql, instance: 'p:9187', member: pg-pve1, env: prod, severity: warning, service: shared-postgresql} - exp_labels: {job: shared-postgresql-sql, instance: 'p:9187', member: pg-pve1, env: prod, severity: warning, service: shared-postgresql}
exp_annotations: exp_annotations:
summary: 共享 PG prod pg-pve1 有事务空闲未提交超过 10 分钟 summary: 共享 PG prod pg-pve1 有事务空闲未提交超过 10 分钟
- name: 主库落在 pve1 持续 15 分钟提示回切
interval: 1m
input_series:
- series: pg_replication_is_replica{job="shared-postgresql-sql",instance="10.60.0.20:9187",cluster="homelab-pg-prod",member="pg-pve1",env="prod"}
values: '0x20'
- series: pg_replication_is_replica{job="shared-postgresql-sql",instance="192.168.10.127:9187",cluster="homelab-pg-prod",member="pg-laptop",env="prod"}
values: '1x20'
alert_rule_test:
- eval_time: 10m
alertname: SharedPostgresPrimaryOnStandbyHost
exp_alerts: []
- eval_time: 16m
alertname: SharedPostgresPrimaryOnStandbyHost
exp_alerts:
- exp_labels: {job: shared-postgresql-sql, instance: '10.60.0.20:9187', cluster: homelab-pg-prod, member: pg-pve1, env: prod, severity: warning, service: shared-postgresql}
exp_annotations:
summary: 共享 PG 生产主库运行在 pg-pve1(HDD),请确认切换原因并计划回切 laptop
- name: 主库在 laptop 不提示
interval: 1m
input_series:
- series: pg_replication_is_replica{job="shared-postgresql-sql",instance="10.60.0.20:9187",member="pg-pve1",env="prod"}
values: '1x20'
alert_rule_test:
- eval_time: 16m
alertname: SharedPostgresPrimaryOnStandbyHost
exp_alerts: []