kubernetes/apps/kopiur-system/kopiur/app/prometheusrule.yaml
1
---
2
# yaml-language-server: $schema=https://prometheus-operator.dev/doc/api/v1/#monitoring.coreos.com/v1.PrometheusRule
3
apiVersion: monitoring.coreos.com/v1
4
kind: PrometheusRule
5
metadata:
6
name: kopiur-backup-integrity
7
spec:
8
groups:
9
- name: kopiur-system.backup-integrity
10
interval: 1m
11
rules:
12
- alert: KopiurBackupSizeCollapsed
13
expr: >-
14
(
15
max by (namespace, policy) (kopiur_policy_last_backup_size_bytes)
16
/
17
max by (namespace, policy) (max_over_time(kopiur_policy_last_backup_size_bytes[7d]))
18
< 0.5
19
)
20
and
21
(
22
max by (namespace, policy) (max_over_time(kopiur_policy_last_backup_size_bytes[7d]))
23
> 10485760
24
)
25
for: 15m
26
labels:
27
severity: critical
28
annotations:
29
summary: >-
30
Backup for {{ $labels.namespace }}/{{ $labels.policy }} collapsed to
31
{{ $value | humanizePercentage }} of its 7-day peak
32
description: >-
33
The newest kopiur snapshot is less than half the size of the largest
34
in the past week, which means the source PVC lost data rather than
35
the backup failing. ACT NOW, BEFORE INVESTIGATING: retention is
36
keepHourly 24 / keepDaily 7, and the daily slot keeps the LAST
37
snapshot of each day, so an evening data loss leaves only the
38
post-loss copy once the hourly window rolls past. Pin the newest
39
good snapshot first with
40
`kubectl patch kopiasnap -n <ns> <snap> --type=merge -p
41
'{"spec":{"pin":true,"deletionPolicy":"Retain"}}'`, and pin one
42
older fallback as well. Then find the cliff with
43
`kubectl get kopiasnap -A -o json | jq -r '.items[] |
44
[.metadata.creationTimestamp, .metadata.name,
45
(.status.stats.sizeBytes//0)] | @tsv'`.
46
- alert: KopiurBackupStale
47
expr: >-
48
(time() - max by (namespace, policy) (kopiur_policy_last_backup_success_timestamp_seconds))
49
> 21600
50
for: 30m
51
labels:
52
severity: warning
53
annotations:
54
summary: >-
55
No successful kopiur backup for {{ $labels.namespace }}/{{ $labels.policy }}
56
in {{ $value | humanizeDuration }}
57
description: >-
58
Schedules run roughly every two hours, so six hours without a
59
success means the policy is wedged, not idle. A stalled policy is
60
silent until the day it is needed, and it removes the only recovery
61
path from a PVC wipe. Check the Snapshot CRs for StagingTimedOut
62
and the miroir agents for frozen DRBD devices.