Add runbooks full capacity alerts#1780
Conversation
📝 WalkthroughWalkthroughAdded Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Important Action Needed: IP Allowlist UpdateIf your organization protects your Git platform with IP whitelisting, please add the new CodeRabbit IP address to your allowlist:
Failure to add the new IP will result in interrupted reviews. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
deploy/internal/prometheus-rules.yaml (1)
213-225:runbook_urlmust be moved inside theannotationsblock—it is not a valid top-level alerting rule field.The PrometheusRule CRD schema only recognizes
alert,record,expr,for,keep_firing_for,labels, andannotationsas top-level rule fields.runbook_urlis a convention for theannotationsmap. As currently placed at the top level, the field will be silently ignored by Kubernetes, and the runbook links will not reach AlertManager.Apply the same fix to all three alerts (lines 225, 238, and 250): move
runbook_urlinto theannotationsblock.🐛 Proposed fix for NooBaaSystemCapacityWarning85
- alert: NooBaaSystemCapacityWarning85 annotations: description: A NooBaa system is approaching its capacity, usage is more than 85% message: A NooBaa System Is Approaching Its Capacity + runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning85.md severity_level: warning storage_type: NooBaa expr: | NooBaa_system_capacity > 85 for: 5m labels: severity: warning - runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning85.md🐛 Proposed fix for NooBaaSystemCapacityWarning95
- alert: NooBaaSystemCapacityWarning95 annotations: description: A NooBaa system is approaching its capacity, usage is more than 95% message: A NooBaa System Is Approaching Its Capacity + runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning95.md severity_level: critical storage_type: NooBaa expr: | NooBaa_system_capacity > 95 for: 5m labels: severity: critical - runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning95.md🐛 Proposed fix for NooBaaSystemCapacityWarning100
- alert: NooBaaSystemCapacityWarning100 annotations: description: A NooBaa system approached its capacity, usage is at 100% message: A NooBaa System Approached Its Capacity + runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning100.md severity_level: critical storage_type: NooBaa expr: | NooBaa_system_capacity == 100 for: 5m labels: severity: critical - runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning100.mdpkg/bundle/deploy.go (1)
4915-4952: Moverunbook_urlintoannotations— top-level placement violates PrometheusRule CRD schema.PrometheusRule CRD alert rules only accept
annotations,labels, and standard fields likeexpr,for,alert. Top-levelrunbook_urlis not a recognized field and will be rejected by the CRD validator. Moverunbook_urlinto theannotationsmap for each alert where it will be properly recognized as a convention for runbook links.✅ Suggested fix
- alert: NooBaaSystemCapacityWarning85 annotations: description: A NooBaa system is approaching its capacity, usage is more than 85% message: A NooBaa System Is Approaching Its Capacity severity_level: warning storage_type: NooBaa + runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning85.md expr: | NooBaa_system_capacity > 85 for: 5m labels: severity: warning - runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning85.md - alert: NooBaaSystemCapacityWarning95 annotations: description: A NooBaa system is approaching its capacity, usage is more than 95% message: A NooBaa System Is Approaching Its Capacity severity_level: critical storage_type: NooBaa + runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning95.md expr: | NooBaa_system_capacity > 95 for: 5m labels: severity: critical - runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning95.md - alert: NooBaaSystemCapacityWarning100 annotations: description: A NooBaa system approached its capacity, usage is at 100% message: A NooBaa System Approached Its Capacity severity_level: critical storage_type: NooBaa + runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning100.md expr: | NooBaa_system_capacity == 100 for: 5m labels: severity: critical - runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/NooBaaSystemCapacityWarning100.md
Signed-off-by: jackyalbo <jacky.albo@gmail.com>
Explain the changes
See runbook markdown in this PR Add NooBaa (MCG) Capacity Alerts openshift/runbooks#380
When alerts appear, a link to the markdown will be available for the customer
Issues: Fixed #xxx / Gap #xxx
Testing Instructions:
Summary by CodeRabbit