修正共享 PG 计数器告警在新序列上的误报
yaml / yaml (pull_request) Successful in 31s

MetricsQL 的 increase() 把新序列首个样本视为从 0 增长:#173 合并后 exporter
首次被抓取,laptop 上 2026-09-25 搭建期累计的 9 次归档失败被当成刚发生,
触发 SharedPostgresWalArchiveFailing。归档本身正常(切换后 .partial 已归档)。

归档失败、死锁、checkpoint 三条改为与 offset 相减:新序列返回空,
计数器重置得负值也不告警。promtool 按 Prometheus 语义执行,测不出这一差异,
新表达式已在 VictoriaMetrics 现网数据上核对为空;补充新序列与重置回归场景。

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
2026-10-01 16:41:04 +00:00
co-authored by Claude Opus 5.5
parent 2c7dd224cf
commit 2b382e1579
2 changed files with 44 additions and 5 deletions
@@ -56,8 +56,13 @@ spec:
labels: {severity: warning, service: shared-postgresql} labels: {severity: warning, service: shared-postgresql}
annotations: annotations:
summary: '共享 PG {{ $labels.env }} 复制槽 {{ $labels.slot_name }} 保留 WAL 超过 1 GiB' summary: '共享 PG {{ $labels.env }} 复制槽 {{ $labels.slot_name }} 保留 WAL 超过 1 GiB'
# 计数器规则不用 increase():MetricsQL 把新序列首个样本视为从 0 增长,exporter 新上线或
# 角色变化时,历史累计值(如 2026-09-25 搭建期的 9 次归档失败)会被当成刚发生而误报。
# 与 offset 相减对新序列返回空;计数器重置(重启/stats reset)得负值,同样不告警。
- alert: SharedPostgresWalArchiveFailing - alert: SharedPostgresWalArchiveFailing
expr: increase(pg_stat_archiver_failed_count{job="shared-postgresql-sql"}[15m]) > 0 expr: >-
pg_stat_archiver_failed_count{job="shared-postgresql-sql"}
- pg_stat_archiver_failed_count{job="shared-postgresql-sql"} offset 15m > 0
labels: {severity: warning, service: shared-postgresql} labels: {severity: warning, service: shared-postgresql}
annotations: annotations:
summary: '共享 PG {{ $labels.env }} {{ $labels.member }} WAL 归档失败,PITR 链可能中断' summary: '共享 PG {{ $labels.env }} {{ $labels.member }} WAL 归档失败,PITR 链可能中断'
@@ -105,13 +110,17 @@ spec:
annotations: annotations:
summary: '共享 PG {{ $labels.env }} {{ $labels.member }} 最老事务已运行超过 1 小时' summary: '共享 PG {{ $labels.env }} {{ $labels.member }} 最老事务已运行超过 1 小时'
- alert: SharedPostgresDeadlocks - alert: SharedPostgresDeadlocks
expr: sum without(datname) (increase(pg_stat_database_deadlocks{job="shared-postgresql-sql"}[15m])) > 0 expr: >-
sum without(datname) (pg_stat_database_deadlocks{job="shared-postgresql-sql"}
- pg_stat_database_deadlocks{job="shared-postgresql-sql"} offset 15m) > 0
labels: {severity: warning, service: shared-postgresql} labels: {severity: warning, service: shared-postgresql}
annotations: annotations:
summary: '共享 PG {{ $labels.env }} {{ $labels.member }} 15 分钟内出现死锁' summary: '共享 PG {{ $labels.env }} {{ $labels.member }} 15 分钟内出现死锁'
# 每日 pgBackRest full 会请求一次 checkpoint;每小时多于 4 次通常意味着 max_wal_size 偏小。 # 每日 pgBackRest full 会请求一次 checkpoint;每小时多于 4 次通常意味着 max_wal_size 偏小。
- alert: SharedPostgresCheckpointsRequested - alert: SharedPostgresCheckpointsRequested
expr: increase(pg_stat_checkpointer_num_requested_total{job="shared-postgresql-sql"}[1h]) > 4 expr: >-
pg_stat_checkpointer_num_requested_total{job="shared-postgresql-sql"}
- pg_stat_checkpointer_num_requested_total{job="shared-postgresql-sql"} offset 1h > 4
for: 15m for: 15m
labels: {severity: warning, service: shared-postgresql} labels: {severity: warning, service: shared-postgresql}
annotations: annotations:
@@ -117,9 +117,9 @@ tests:
interval: 1m interval: 1m
input_series: input_series:
- series: pg_stat_database_deadlocks{job="shared-postgresql-sql",instance="p:9187",member="pg-pve1",env="prod",datname="app"} - series: pg_stat_database_deadlocks{job="shared-postgresql-sql",instance="p:9187",member="pg-pve1",env="prod",datname="app"}
values: '0 0 0 1 1 1' values: '0x20 1x5'
alert_rule_test: alert_rule_test:
- eval_time: 5m - eval_time: 22m
alertname: SharedPostgresDeadlocks alertname: SharedPostgresDeadlocks
exp_alerts: exp_alerts:
- exp_labels: {job: shared-postgresql-sql, instance: 'p:9187', member: pg-pve1, env: prod, severity: warning, service: shared-postgresql} - exp_labels: {job: shared-postgresql-sql, instance: 'p:9187', member: pg-pve1, env: prod, severity: warning, service: shared-postgresql}
@@ -163,3 +163,33 @@ tests:
- eval_time: 16m - eval_time: 16m
alertname: SharedPostgresPrimaryOnStandbyHost alertname: SharedPostgresPrimaryOnStandbyHost
exp_alerts: [] exp_alerts: []
- name: 新出现的计数器序列带历史累计值不误报归档失败
interval: 1m
input_series:
- series: pg_stat_archiver_failed_count{job="shared-postgresql-sql",instance="l:9187",member="pg-laptop",env="prod"}
values: '_x20 9x20'
alert_rule_test:
- eval_time: 25m
alertname: SharedPostgresWalArchiveFailing
exp_alerts: []
- name: 归档失败计数增长告警
interval: 1m
input_series:
- series: pg_stat_archiver_failed_count{job="shared-postgresql-sql",instance="l:9187",member="pg-laptop",env="prod"}
values: '9x20 10x5'
alert_rule_test:
- eval_time: 22m
alertname: SharedPostgresWalArchiveFailing
exp_alerts:
- exp_labels: {job: shared-postgresql-sql, instance: 'l:9187', member: pg-laptop, env: prod, severity: warning, service: shared-postgresql}
exp_annotations:
summary: 共享 PG prod pg-laptop WAL 归档失败,PITR 链可能中断
- name: 计数器重置不误报 checkpoint
interval: 1m
input_series:
- series: pg_stat_checkpointer_num_requested_total{job="shared-postgresql-sql",instance="l:9187",member="pg-laptop",env="prod"}
values: '16x60 0x30'
alert_rule_test:
- eval_time: 85m
alertname: SharedPostgresCheckpointsRequested
exp_alerts: []