Adding BWC test for remote publication enabled cluster by gargharsh3134 · Pull Request #20221 · opensearch-project/OpenSearch

gargharsh3134 · 2025-12-12T10:55:05Z

Description

RemotePublication feature was missing BackWardCompatibility rolling upgrade integration tests which was leading to unintended incompatibilities between versions. Added changes in qa rolling upgrades tests to execute a new and the existing tests with and without remote publication enabled.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Summary by CodeRabbit

Tests
- Added backward compatibility tests for remote publication enabled clusters in rolling upgrade scenarios.
- Extended test suite to validate cluster state, templates, and settings across upgrade phases.
- New rolling upgrade test infrastructure supports remote cluster configuration and workflow.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

coderabbitai · 2025-12-12T10:55:11Z

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

📝 Walkthrough

Walkthrough

This PR adds backward compatibility testing for OpenSearch clusters with remote publication enabled. It introduces a new test class, refactors the rolling upgrade build configuration with a helper function for centralized test task setup, and extends the test workflow to include remote cluster testing scenarios alongside existing local cluster tests.

Changes

Cohort / File(s)	Summary
Documentation `CHANGELOG.md`	Adds new Unreleased 3.x entry documenting the backward compatibility test for remote publication enabled clusters.
Build Configuration `qa/rolling-upgrade/build.gradle`	Introduces `configureTestTask` helper function for centralized test task configuration; refactors existing tasks to use the helper; adds remote publication cluster configuration with dedicated test tasks (remoteOldClusterTest, remoteOneThirdUpgradedTest, remoteTwoThirdsUpgradedTest, remoteUpgradedClusterTest); updates top-level BWC task dependency to remote upgraded cluster test; fixes typo in comment ("updgrade" → "upgrade"); adds remote store and state configuration settings.
Test Implementation `qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java`	New test class extending `AbstractRollingTestCase` that validates cluster state across OLD and non-OLD cluster types when remote publication is enabled; creates and validates index templates, component templates, composable templates, and cluster settings; includes helper methods for nested value extraction and remote publication verification.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

patch

Suggested reviewers

dbwiddis
andrross
cwperks
msfroh
mch2
shwetathareja
sachinpkale

Poem

🐰 A rolling test now hops with glee,
Remote publications dance with spree,
Templates twirl through cluster state,
Backward bounds—we celebrate!
Upgrade paths both near and far,
✨ A testing star!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: adding a BWC (backward compatibility) test for remote publication enabled clusters, which aligns with the actual code changes that introduce new test classes and infrastructure.
Description check	✅ Passed	The PR description addresses the main objective and includes most required sections, though the Related Issues section lacks a specific issue number.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2025-12-12T12:16:35Z

✅ Gradle check result for c6ef629: SUCCESS

codecov · 2025-12-12T12:17:42Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.26%. Comparing base (be7b387) to head (5660fca).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #20221      +/-   ##
============================================
- Coverage     73.28%   73.26%   -0.02%     
- Complexity    71825    71896      +71     
============================================
  Files          5793     5793              
  Lines        328844   328846       +2     
  Branches      47343    47343              
============================================
- Hits         240978   240930      -48     
- Misses        68571    68617      +46     
- Partials      19295    19299       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rajiv-kv

Can you include the build logs (from your local runs / remote jenkins) where the additional tests are able to run successfully with remote publication enabled ?

qa/rolling-upgrade/build.gradle

qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java

qa/rolling-upgrade/build.gradle

github-actions · 2026-01-06T08:35:19Z

❌ Gradle check result for e3183fa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (2)

qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java (2)
85-97: Both persistent and transient settings use identical values.

Both addPersistentSettings() and addTransientSettings() configure the same cluster.remote.cluster.seeds path with the same value. Transient settings take precedence over persistent ones, so the persistent setting is effectively shadowed.

If the intent is to test that both setting types survive the rolling upgrade, consider using different values or different setting keys to distinguish them in verification.

143-150: Transient settings verification is missing.

verifySettingsInClusterState() only validates that persistent settings exist but doesn't verify the transient settings added by addTransientSettings(). Consider adding verification for transient settings to ensure complete coverage.
🔎 Suggested addition
 private static void verifySettingsInClusterState() throws Exception {
     Request getSettingRequest = new Request("GET", "/_cluster/settings");
     Response response = client().performRequest(getSettingRequest);
     assertOK(response);
     Map<String, Object> responseMap = entityAsMap(response);
     Map<String, Object> persistent = (Map<String, Object>) responseMap.get("persistent");
     assertNotNull("Persistent settings should exist in metadata", persistent);
+    Map<String, Object> transient_ = (Map<String, Object>) responseMap.get("transient");
+    assertNotNull("Transient settings should exist in metadata", transient_);
 }

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 66d2482 and e3183fa.

📒 Files selected for processing (3)

CHANGELOG.md
qa/rolling-upgrade/build.gradle
qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java

🔇 Additional comments (9)

CHANGELOG.md (1)

13-13: LGTM!

The changelog entry is correctly placed under the "Added" section and follows the project's formatting conventions.

qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java (4)

19-24: LGTM!

Class structure is appropriate. The @SuppressWarnings("unchecked") at class level is justified given the pervasive use of Map<String, Object> casts throughout the verification methods.

50-61: Consider verifying cluster settings after upgrade.

The verifySettingsInClusterState() call is missing in the non-OLD branch. If persistent settings added during the OLD cluster phase should survive the rolling upgrade, consider adding this verification to ensure settings persistence is maintained across versions.

If the omission is intentional (e.g., settings are expected to change or reset), a comment explaining this would be helpful.

64-83: LGTM!

Template creation methods are well-structured. The composable template correctly references the component template, establishing the expected relationship for verification.

152-170: LGTM!

The helper methods are well-implemented. getNestedValue() handles edge cases gracefully, and verifyRemotePublicationEnabled() correctly verifies the setting from cluster defaults.

qa/rolling-upgrade/build.gradle (4)

40-55: LGTM!

The configureTestTask helper function effectively centralizes test task configuration, reducing duplication. The closure delegation pattern (extraConfig.delegate = delegate) is the idiomatic Groovy approach for extending configuration blocks.

84-113: LGTM!

The local cluster test tasks are correctly refactored to use the helper function. Setting excludeRemoteTest=true appropriately excludes RemotePublicationClusterStateIT from running against the non-remote-publication-enabled cluster.

115-132: LGTM!

The remote publication cluster is properly configured with the necessary settings to enable remote state store and publication. The filesystem-based repository under buildDir ensures test isolation.

134-166: LGTM!

The remote cluster test tasks correctly mirror the local cluster workflow. The dependency chain ensures:

Local cluster tests complete first (with RemotePublicationClusterStateIT excluded)

Remote cluster tests run with RemotePublicationClusterStateIT included

The bwc task depends on the full chain completing

This properly validates remote publication backward compatibility.

rajiv-kv

LGTM ! Please fix the build errors.

github-actions · 2026-01-08T05:14:44Z

❌ Gradle check result for a89ec02: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java

Signed-off-by: Harsh Garg <gkharsh@amazon.com>

github-actions · 2026-01-19T06:20:47Z

✅ Gradle check result for 5660fca: SUCCESS

…oject#20221) * Adding BWC test for remote publication enabled cluster Signed-off-by: Harsh Garg <gkharsh@amazon.com>

…earch-project#20221)" The remote cluster state feature does not support forward compatibility by design. Therefore these rolling upgrade tests are not compatible with the feature because they make no guarantees whether an old-version or new-version node will be elected as cluster manager in the mixed version state. Reverting this commit and will let the remote cluster state experts come back with a non-flaky test. This reverts commit dbb6d2e. Signed-off-by: Andrew Ross <andrross@amazon.com>

rajiv-kv reviewed Dec 23, 2025

View reviewed changes

qa/rolling-upgrade/build.gradle Outdated Show resolved Hide resolved

qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java Show resolved Hide resolved

qa/rolling-upgrade/build.gradle Show resolved Hide resolved

gargharsh3134 force-pushed the remoteBWCTest branch 2 times, most recently from 4bb7fe0 to e3183fa Compare January 6, 2026 08:29

gargharsh3134 marked this pull request as ready for review January 6, 2026 13:40

gargharsh3134 requested a review from a team as a code owner January 6, 2026 13:40

coderabbitai bot reviewed Jan 6, 2026

View reviewed changes

rajiv-kv reviewed Jan 7, 2026

View reviewed changes

shwetathareja reviewed Jan 8, 2026

View reviewed changes

qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java Show resolved Hide resolved

andrross reviewed Jan 8, 2026

View reviewed changes

qa/rolling-upgrade/src/test/java/org/opensearch/upgrades/RemotePublicationClusterStateIT.java Outdated Show resolved Hide resolved

Harsh Garg added 2 commits January 19, 2026 09:28

Adding BWC test for remote publication enabled cluster

3c4e9cc

Signed-off-by: Harsh Garg <gkharsh@amazon.com>

Retry Build

177f806

Signed-off-by: Harsh Garg <gkharsh@amazon.com>

gargharsh3134 force-pushed the remoteBWCTest branch 2 times, most recently from 26cfb71 to aed21ab Compare January 19, 2026 04:28

Updating documentation for new remote publication tests

5660fca

Signed-off-by: Harsh Garg <gkharsh@amazon.com>

gargharsh3134 force-pushed the remoteBWCTest branch from aed21ab to 5660fca Compare January 19, 2026 04:42

shwetathareja approved these changes Jan 20, 2026

View reviewed changes

shwetathareja merged commit dbb6d2e into opensearch-project:main Jan 20, 2026
34 checks passed

This was referenced Feb 18, 2026

[AUTOCUT] Gradle Check Flaky Test Report for ClientYamlTestSuiteIT #14319

Open

[AUTOCUT] Gradle Check Flaky Test Report for Gradle Test Run :modules:parent-join:internalClusterTest #20662

Open

opensearch-ci-bot mentioned this pull request Mar 9, 2026

[AUTOCUT] Gradle Check Flaky Test Report for Gradle Test Run :server:test #19729

Closed

andrross mentioned this pull request Mar 18, 2026

[BUG] Remote cluster state compatibility failures #20910

Closed

This was referenced Mar 20, 2026

Revert "Adding BWC test for remote publication enabled cluster" #20942

Merged

[BUG] Remote cluster state backward compatibility tests #20948

Open

Conversation

gargharsh3134 commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

Poem

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

codecov bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rajiv-kv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 6, 2026

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

rajiv-kv left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gargharsh3134 commented Dec 12, 2025 •

edited

Loading

coderabbitai bot commented Dec 12, 2025 •

edited

Loading

codecov bot commented Dec 12, 2025 •

edited

Loading