Added prometheus alert rules for DB volume size capacity#1660
Added prometheus alert rules for DB volume size capacity#1660dannyzaken merged 1 commit intonoobaa:masterfrom
Conversation
WalkthroughA new Prometheus alert rule group named Changes
Sequence Diagram(s)sequenceDiagram
participant Prometheus
participant NooBaa DB Pod 1
participant NooBaa DB Pod 2
participant Alertmanager
loop Every scrape interval
Prometheus->>NooBaa DB Pod 1: Query cnpg_pg_database_size_bytes
Prometheus->>NooBaa DB Pod 1: Query kube_persistentvolumeclaim_resource_requests_storage_bytes
Prometheus->>NooBaa DB Pod 2: Query cnpg_pg_database_size_bytes
Prometheus->>NooBaa DB Pod 2: Query kube_persistentvolumeclaim_resource_requests_storage_bytes
Prometheus->>Prometheus: Evaluate alert rules for both pods
alt Usage > 80% or 90% for 5m
Prometheus->>Alertmanager: Fire warning/critical alert for respective pod
end
end
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
deploy/internal/prometheus-rules.yaml (4)
252-255: Drop the duplicated word inmessageannotation.
messagecurrently reads:
NooBaa NooBaa database …The double “NooBaa” looks like a copy-paste artefact and shows up verbatim in Alertmanager UIs.
- message: NooBaa NooBaa database on pod noobaa-db-pg-cluster-1 is using 80% of its PVC capacity. + message: NooBaa database on pod noobaa-db-pg-cluster-1 is using 80% of its PVC capacity.
265-279: Alert duplication can be collapsed into a single parametrised rule.You define two rules with identical alert name/logic, differing only by the hard-coded pod/PVC names.
Maintaining multiple near-identical rules is error-prone (threshold drift, doc fixes, etc.).Consider a single rule that covers both replicas by using a regex filter and relying on the
pod/persistentvolumeclaimlabels to disambiguate:- - alert: NooBaaDatabaseReachingCapacity - ... - cnpg_pg_database_size_bytes{..., pod="noobaa-db-pg-cluster-1"} - /on(namespace) group_left(persistentvolumeclaim) - kube_persistentvolumeclaim_resource_requests_storage_bytes{..., persistentvolumeclaim="noobaa-db-pg-cluster-1"} - ) * 100 > 80 + - alert: NooBaaDatabaseReachingCapacity + ... + cnpg_pg_database_size_bytes{datname="nbcore", namespace="openshift-storage", pod=~"noobaa-db-pg-cluster-[12]"} + /on(namespace) group_left(persistentvolumeclaim) + kube_persistentvolumeclaim_resource_requests_storage_bytes{namespace="openshift-storage", persistentvolumeclaim=~"noobaa-db-pg-cluster-[12]"} + ) * 100 > 80You’d keep only one rule; Alertmanager will fire one alert per distinct
podlabel value.Same applies to the critical (>90 %) rule below.
Reduces duplication while preserving functionality.
282-285: Same duplicated “NooBaa NooBaa” typo in critical alert message.Apply the same one-word removal as suggested for the warning rule.
296-309: Consider consolidating critical (>90 %) rules in the same way as the warning rules.The two critical rules are also identical except for pod/PVC.
Consolidating them (regex or template) will halve maintenance overhead.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
deploy/internal/prometheus-rules.yaml(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: Neon-White
PR: noobaa/noobaa-operator#1587
File: pkg/system/reconciler.go:287-291
Timestamp: 2025-07-14T10:59:21.959Z
Learning: NooBaa operator design constraint: Multiple NooBaa instances are not supported in the same namespace, only across different namespaces. This means hard-coded resource names like ConfigMaps are acceptable within a namespace scope.
deploy/internal/prometheus-rules.yaml (1)
Learnt from: Neon-White
PR: noobaa/noobaa-operator#1587
File: pkg/system/reconciler.go:287-291
Timestamp: 2025-07-14T10:59:21.959Z
Learning: NooBaa operator design constraint: Multiple NooBaa instances are not supported in the same namespace, only across different namespaces. This means hard-coded resource names like ConfigMaps are acceptable within a namespace scope.
🔇 Additional comments (1)
deploy/internal/prometheus-rules.yaml (1)
257-262: Join key mismatch could yield wrong ratios if more PVCs exist.The vector matching uses
/ on(namespace) group_left(persistentvolumeclaim).Because the left-hand metric is filtered by
pod(notpersistentvolumeclaim), multiple PVCs in the same namespace would match and the group-left will pick an arbitrary one.
Safer to join on bothnamespaceand a deterministic key common to both sides – here the PVC name:/on (namespace, persistentvolumeclaim)and prefix the left metric with
persistentvolumeclaim="<same-name>"label vialabel_replace, or switch the LHS filter topersistentvolumeclaim.
Please verify the cardinality to avoid silent mis-calculation.
Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
deploy/internal/prometheus-rules.yaml(1 hunks)pkg/bundle/deploy.go(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- deploy/internal/prometheus-rules.yaml
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: Neon-White
PR: noobaa/noobaa-operator#1587
File: pkg/system/reconciler.go:287-291
Timestamp: 2025-07-14T10:59:21.959Z
Learning: NooBaa operator design constraint: Multiple NooBaa instances are not supported in the same namespace, only across different namespaces. This means hard-coded resource names like ConfigMaps are acceptable within a namespace scope.
pkg/bundle/deploy.go (1)
undefined
<retrieved_learning>
Learnt from: Neon-White
PR: #1587
File: pkg/system/reconciler.go:287-291
Timestamp: 2025-07-14T10:59:21.959Z
Learning: NooBaa operator design constraint: Multiple NooBaa instances are not supported in the same namespace, only across different namespaces. This means hard-coded resource names like ConfigMaps are acceptable within a namespace scope.
</retrieved_learning>
🔇 Additional comments (1)
pkg/bundle/deploy.go (1)
4540-4540: LGTM: SHA256 checksum correctly updated.The checksum update reflects the addition of the new Prometheus alert rules in the embedded YAML content.
| - name: noobaa-db-alert.rules | ||
| rules: | ||
| - alert: NooBaaDatabaseReachingCapacity | ||
| annotations: | ||
| description: The NooBaa database on pod noobaa-db-pg-cluster-1 is using 80% of its PVC requested size. | ||
| message: NooBaa NooBaa database on pod noobaa-db-pg-cluster-1 is using 80% of its PVC capacity. | ||
| severity_level: warning | ||
| storage_type: NooBaa | ||
| expr: | | ||
| ( | ||
| cnpg_pg_database_size_bytes{datname="nbcore", namespace="openshift-storage", pod="noobaa-db-pg-cluster-1"} | ||
| /on(namespace) group_left(persistentvolumeclaim) | ||
| kube_persistentvolumeclaim_resource_requests_storage_bytes{namespace="openshift-storage", persistentvolumeclaim="noobaa-db-pg-cluster-1"} | ||
| ) * 100 > 80 | ||
| for: 5m | ||
| labels: | ||
| severity: warning | ||
| - alert: NooBaaDatabaseReachingCapacity | ||
| annotations: | ||
| description: The NooBaa database on pod noobaa-db-pg-cluster-2 is using 80% of its PVC requested size. | ||
| message: NooBaa NooBaa database on pod noobaa-db-pg-cluster-2 is using 80% of its PVC capacity. | ||
| severity_level: warning | ||
| storage_type: NooBaa | ||
| expr: | | ||
| ( | ||
| cnpg_pg_database_size_bytes{datname="nbcore", namespace="openshift-storage", pod="noobaa-db-pg-cluster-2"} | ||
| /on(namespace) group_left(persistentvolumeclaim) | ||
| kube_persistentvolumeclaim_resource_requests_storage_bytes{namespace="openshift-storage", persistentvolumeclaim="noobaa-db-pg-cluster-2"} | ||
| ) * 100 > 80 | ||
| for: 5m | ||
| labels: | ||
| severity: warning | ||
| - alert: NooBaaDatabaseStorageFull | ||
| annotations: | ||
| description: The NooBaa database on pod noobaa-db-pg-cluster-1 is using over 90% of its PVC requested size. Increase the DB size as soon as possible. | ||
| message: NooBaa NooBaa database on pod noobaa-db-pg-cluster-1 is using over 90% of its PVC capacity. | ||
| severity_level: critical | ||
| storage_type: NooBaa | ||
| expr: | | ||
| ( | ||
| cnpg_pg_database_size_bytes{datname="nbcore", namespace="openshift-storage", pod="noobaa-db-pg-cluster-1"} | ||
| /on(namespace) group_left(persistentvolumeclaim) | ||
| kube_persistentvolumeclaim_resource_requests_storage_bytes{namespace="openshift-storage", persistentvolumeclaim="noobaa-db-pg-cluster-1"} | ||
| ) * 100 > 90 | ||
| for: 5m | ||
| labels: | ||
| severity: critical | ||
| - alert: NooBaaDatabaseStorageFull | ||
| annotations: | ||
| description: The NooBaa database on pod noobaa-db-pg-cluster-2 is using over 90% of its PVC requested size. Increase the DB size as soon as possible. | ||
| message: NooBaa NooBaa database on pod noobaa-db-pg-cluster-2 is using over 90% of its PVC capacity. | ||
| severity_level: critical | ||
| storage_type: NooBaa | ||
| expr: | | ||
| ( | ||
| cnpg_pg_database_size_bytes{datname="nbcore", namespace="openshift-storage", pod="noobaa-db-pg-cluster-2"} | ||
| /on(namespace) group_left(persistentvolumeclaim) | ||
| kube_persistentvolumeclaim_resource_requests_storage_bytes{namespace="openshift-storage", persistentvolumeclaim="noobaa-db-pg-cluster-2"} | ||
| ) * 100 > 90 | ||
| for: 5m | ||
| labels: | ||
| severity: critical |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Verify metric availability and consider deployment flexibility.
The Prometheus alert rules are well-structured with appropriate thresholds (80% warning, 90% critical) and evaluation periods. The expressions correctly use Prometheus join syntax to calculate storage usage percentages.
However, consider the following:
-
Hard-coded pod names: The alerts target specific pods (
noobaa-db-pg-cluster-1andnoobaa-db-pg-cluster-2). While this aligns with NooBaa's single-instance design, ensure these pod names remain consistent across deployments. -
Metric dependencies: The alerts rely on
cnpg_pg_database_size_bytesandkube_persistentvolumeclaim_resource_requests_storage_bytesmetrics. Verify these metrics are available in the target environment.
🏁 Script executed:
#!/bin/bash
# Description: Verify the existence of required Prometheus metrics and check for any existing references to the hard-coded pod names.
# Expected: Find references to the pod names and confirm metric usage patterns.
echo "=== Checking for references to hard-coded pod names ==="
rg -A 3 -B 3 "noobaa-db-pg-cluster-[12]"
echo "=== Checking for other Prometheus metric references ==="
rg -A 2 -B 2 "cnpg_pg_database_size_bytes|kube_persistentvolumeclaim_resource_requests_storage_bytes"
echo "=== Checking for existing alert rule patterns ==="
rg -A 5 -B 5 "PrometheusRule|alert:"Length of output: 50433
Replace hard-coded pod names with a regex selector and verify metric availability
To ensure these alerts remain accurate as your Postgres cluster scales or pod names change, and to guarantee they fire correctly, please:
• Update both pkg/bundle/deploy.go (and the embedded File_deploy_internal_prometheus_rules_yaml) and deploy/internal/prometheus-rules.yaml
– Replace
{pod="noobaa-db-pg-cluster-1"} and {pod="noobaa-db-pg-cluster-2"}
with a regex selector, e.g.:
pod=~"noobaa-db-pg-cluster-.*"
• Confirm that your Prometheus stack scrapes:
– cnpg_pg_database_size_bytes{datname="nbcore"} (from the CNPG exporter)
– kube_persistentvolumeclaim_resource_requests_storage_bytes (via kube-state-metrics)
• (Optional) Document this metric requirement in your deployment guide to help operators enable or troubleshoot these alerts.
🤖 Prompt for AI Agents
In pkg/bundle/deploy.go around lines 4789 to 4850, replace the hard-coded pod
names "noobaa-db-pg-cluster-1" and "noobaa-db-pg-cluster-2" in the Prometheus
alert expressions with a regex selector like pod=~"noobaa-db-pg-cluster-.*" to
cover all relevant pods dynamically. Also, verify that the Prometheus stack
scrapes the required metrics cnpg_pg_database_size_bytes{datname="nbcore"} and
kube_persistentvolumeclaim_resource_requests_storage_bytes, and optionally
document these metric requirements in the deployment guide for operator
reference.
liranmauda
left a comment
There was a problem hiding this comment.
There are 2 alerts and each for cluster-1 and 2.
Could we have more than those (e.g cluster-3)?
And will we always use the namespace openshift-storage ?
| description: The NooBaa database on pod noobaa-db-pg-cluster-1 is using 80% of its PVC requested size. | ||
| message: NooBaa NooBaa database on pod noobaa-db-pg-cluster-1 is using 80% of its PVC capacity. |
There was a problem hiding this comment.
Is the pod name always noobaa-db-pg-cluster-1 ?
There was a problem hiding this comment.
it's always these names. I tried generalizing with regex, but it just didn't work. I will try again after this PR as a fix.
As for the namespace, I think we can remove it. I will test it and update the PR.
Explain the changes
Issues: Fixed #xxx / Gap #xxx
Testing Instructions:
Summary by CodeRabbit