Exporting replication_target_status metric to use in prometheus alert by aayushchouhan09 · Pull Request #9385 · noobaa/noobaa-core

aayushchouhan09 · 2026-01-20T06:16:25Z

Describe the Problem

Currently, we don't have a metric which shows the status of target bucket reachability.

Explain the Changes

We have exported a metrics NooBaa_replication_target_status which can be used by Prometheus alert.

Issues: Fixed #xxx / Gap #xxx

JIRA: https://issues.redhat.com/browse/RHSTOR-8110
Operator PR: Added a prometheus alert rule for unreachable target noobaa-operator#1778

Testing Instructions:

Check operator PR.

Doc added/updated
Tests added

Summary by CodeRabbit

New Features
- Added monitoring for replication target reachability, exposing a way to update target status so destinations are tracked as reachable or unreachable.
Bug Fixes
- Replication scanning and sync flows now update target reachability on success and on errors, improving status accuracy and reliability.
- Added a utility to programmatically set replication target reachability.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

coderabbitai · 2026-01-20T06:16:39Z

📝 Walkthrough

Walkthrough

Adds a Prometheus Gauge replication_target_status and a public setter; introduces a utility to update it; and integrates explicit reachable/unreachable updates and error handling into replication scanner flows.

Changes

Cohort / File(s)	Summary
Metric Definition `src/server/analytic_services/prometheus_reports/noobaa_core_report.js`	Added `replication_target_status` Gauge with `source_bucket` and `target_bucket` labels and `set_replication_target_status(source, target, is_reachable)` method.
Replication Status Utility `src/server/utils/replication_utils.js`	Added `update_replication_target_status(source_bucket, target_bucket, is_reachable)` to unwrap bucket names and call the new metric setter; exported the function.
Replication Scanners `src/server/bg_services/replication_scanner.js`, `src/server/bg_services/log_replication_scanner.js`	On missing destination bucket or errors during bucket-diff / head-objects processing, mark target unreachable; on successful diff/processing mark reachable; added try-catch paths and explicit status updates.

Sequence Diagram(s)

sequenceDiagram
    participant Scanner as Replication Scanner
    participant Utils as Replication Utils
    participant Report as NooBaa Core Report
    participant Prometheus as Prometheus

    Scanner->>Scanner: Resolve source/destination buckets
    alt Destination bucket missing
        Scanner->>Utils: update_replication_target_status(source, dest_id, false)
        Utils->>Report: set_replication_target_status(source, dest_id, 0)
        Report->>Prometheus: Gauge.set(0)
    else Perform diff / head-objects
        Scanner->>Report: attempt bucket-diff / head-objects
        alt Ops succeed
            Scanner->>Utils: update_replication_target_status(source, dest, true)
            Utils->>Report: set_replication_target_status(source, dest, 1)
            Report->>Prometheus: Gauge.set(1)
        else Ops fail
            Scanner->>Utils: update_replication_target_status(source, dest, false)
            Utils->>Report: set_replication_target_status(source, dest, 0)
            Report->>Prometheus: Gauge.set(0)
        end
    end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

size/M

Suggested reviewers

liranmauda
tangledbytes
naveenpaul1

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: exporting a new replication_target_status metric for Prometheus alerts, which is clearly reflected in all modified files.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

alphaprinz

One comment about a swallowed exception, otherwise LGTM.

src/server/bg_services/replication_scanner.js

Signed-off-by: Aayush Chouhan <achouhan@redhat.com>

coderabbitai

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@src/server/bg_services/log_replication_scanner.js`:
- Around line 168-194: The code currently marks the target reachable after
iterating candidates regardless of whether bucketDiff.get_buckets_diff was ever
called; introduce a boolean flag (e.g., diffTried or diffAttempted) before the
for-loop, set it to true right before/when bucketDiff.get_buckets_diff is
invoked inside the loop (the block that produces keys_diff_map and merges into
diff_keys), and only call
replication_utils.update_replication_target_status(src_bucket.name,
dst_bucket.name, true) after the loop if that flag is true; keep the existing
catch behavior (marking unreachable and rethrowing) unchanged so an exception
during get_buckets_diff still marks the target unreachable.

src/server/bg_services/log_replication_scanner.js

aayushchouhan09 requested review from a team, alphaprinz, liranmauda, naveenpaul1 and tangledbytes and removed request for a team January 20, 2026 06:16

pull-request-size bot added the size/L label Jan 20, 2026

aayushchouhan09 mentioned this pull request Jan 20, 2026

Added a prometheus alert rule for unreachable target noobaa/noobaa-operator#1778

Merged

2 tasks

alphaprinz approved these changes Jan 20, 2026

View reviewed changes

src/server/bg_services/replication_scanner.js Outdated Show resolved Hide resolved

aayushchouhan09 force-pushed the targ-alert branch from c433773 to 8d06cad Compare January 21, 2026 09:17

Exporting replication_target_status metric to use in prometheus alert

5be2dce

Signed-off-by: Aayush Chouhan <achouhan@redhat.com>

aayushchouhan09 force-pushed the targ-alert branch from 8d06cad to 5be2dce Compare January 22, 2026 11:50

coderabbitai bot reviewed Jan 22, 2026

View reviewed changes

src/server/bg_services/log_replication_scanner.js Show resolved Hide resolved

aayushchouhan09 merged commit b859495 into noobaa:master Jan 23, 2026
18 of 19 checks passed

aayushchouhan09 deleted the targ-alert branch January 23, 2026 17:24

This was referenced Jan 27, 2026

Added bucket_mode as labels in metrics - bucket_status and namespace_bucket_status #9386

Merged

prometheus alerts - replace bucket capacity with BS capacity DF3671 #9334

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exporting replication_target_status metric to use in prometheus alert#9385

Exporting replication_target_status metric to use in prometheus alert#9385
aayushchouhan09 merged 1 commit intonoobaa:masterfrom
aayushchouhan09:targ-alert

aayushchouhan09 commented Jan 20, 2026 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Jan 20, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Suggested labels

Suggested reviewers

Uh oh!

alphaprinz left a comment

Uh oh!

Uh oh!

coderabbitai bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

aayushchouhan09 commented Jan 20, 2026 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the Problem

Explain the Changes

Issues: Fixed #xxx / Gap #xxx

Testing Instructions:

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Suggested labels

Suggested reviewers

Uh oh!

alphaprinz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aayushchouhan09 commented Jan 20, 2026 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Jan 20, 2026 •

edited

Loading