Skip to content

Fixing flaky test in ClusterManagerDisruptionIT#16992

Open
jaideep-m wants to merge 1 commit intoopensearch-project:mainfrom
jaideep-m:flaky-test-fix
Open

Fixing flaky test in ClusterManagerDisruptionIT#16992
jaideep-m wants to merge 1 commit intoopensearch-project:mainfrom
jaideep-m:flaky-test-fix

Conversation

@jaideep-m
Copy link
Copy Markdown

@jaideep-m jaideep-m commented Jan 10, 2025

Description

Fixes Flakiness of IT - testIsolateClusterManagerAndVerifyClusterStateConsensus in ClusterManagerDisruptionIT

The current test verifies

1. Cluster State Updates:

  • After a network partition is healed, the cluster will attempt to reconcile the states of all nodes.
  • However, the process of updating the cluster state is asynchronous and depends on various factors.

2. Failed Updates:

  • The assertion clusterStateStats.getUpdateFailed() > 0 assumes that there will always be failed cluster state updates on the previously isolated node.
  • This assumption may not always hold true, especially if: a) The cluster reconciles quickly without conflicts. b) The timing of the check happens after successful reconciliation.

The proposed approach is better for the following reasons:

1. Broader Coverage:

  • It checks for any kind of cluster state activity, not just failed updates.
  • This can catch scenarios where the cluster state changed successfully or where time was spent on updates without necessarily failing.

2. Reduced Flakiness:

  • The original test might fail if the cluster manages to reconcile without any failed updates, which could happen in some scenarios.
  • The new approach will pass if there's any indication of cluster state activity, reducing false negatives.

3. Enhanced Assertion Logic:

  • Previously: Only checked for failed cluster state updates.
  • Now: Verifies any cluster state activity (failed updates, successful updates, or time spent on updates).

4. Improved Logging:

  • Added detailed logging of cluster state statistics for better diagnostics and debugging.

5. Timeout Adjustment:

  • Implemented assertBusy with a 30-second timeout to allow sufficient time for cluster state changes to occur and be detected.

Related Issues

Resolves #[12095]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 10, 2025

❌ Gradle check result for 99001ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 10, 2025

❌ Gradle check result for 99001ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

This was referenced Jan 11, 2025
Open
Closed
@jaideep-m
Copy link
Copy Markdown
Author

jaideep-m commented Jan 13, 2025 •
edited
Loading

https://build.ci.opensearch.org/job/gradle-check/52078/

Gradle check is failing due to

[Test Result](https://build.ci.opensearch.org/job/gradle-check/52078/testReport/) (4 failures / +3)
[org.opensearch.indices.replication.SegmentReplicationAllocationIT.testAllocationAndRebalanceWithDisruption](https://build.ci.opensearch.org/job/gradle-check/52078/testReport/junit/org.opensearch.indices.replication/SegmentReplicationAllocationIT/testAllocationAndRebalanceWithDisruption/)
[org.opensearch.indices.replication.SegmentReplicationAllocationIT.testAllocationAndRebalanceWithDisruption](https://build.ci.opensearch.org/job/gradle-check/52078/testReport/junit/org.opensearch.indices.replication/SegmentReplicationAllocationIT/testAllocationAndRebalanceWithDisruption_2/)
[org.opensearch.indices.replication.SegmentReplicationAllocationIT.testAllocationAndRebalanceWithDisruption](https://build.ci.opensearch.org/job/gradle-check/52078/testReport/junit/org.opensearch.indices.replication/SegmentReplicationAllocationIT/testAllocationAndRebalanceWithDisruption_3/)
[org.opensearch.indices.replication.SegmentReplicationAllocationIT.testAllocationAndRebalanceWithDisruption](https://build.ci.opensearch.org/job/gradle-check/52078/testReport/junit/org.opensearch.indices.replication/SegmentReplicationAllocationIT/testAllocationAndRebalanceWithDisruption_4/)

This is due to a known flaky test issue - #14327

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

opensearch-trigger-bot bot commented Feb 12, 2025

This PR is stalled because it has been open for 30 days with no activity.

@opensearch-trigger-bot opensearch-trigger-bot bot added the stalled Issues that have stalled label Feb 12, 2025
@andrross
Copy link
Copy Markdown
Member

andrross commented Aug 27, 2025

@jaideep-m Can you rebase this PR and also fix the DCO error by adding the signed off by tag to your commit?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anasalkouz anasalkouz Awaiting requested review from anasalkouz

@andrross andrross Awaiting requested review from andrross

@ashking94 ashking94 Awaiting requested review from ashking94

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis

@gbbafna gbbafna Awaiting requested review from gbbafna

@jainankitk jainankitk Awaiting requested review from jainankitk

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal

@linuxpi linuxpi Awaiting requested review from linuxpi

@mch2 mch2 Awaiting requested review from mch2

@msfroh msfroh Awaiting requested review from msfroh

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19

@reta reta Awaiting requested review from reta

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159

@sachinpkale sachinpkale Awaiting requested review from sachinpkale

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli

@shwetathareja shwetathareja Awaiting requested review from shwetathareja

@sohami sohami Awaiting requested review from sohami

@VachaShah VachaShah Awaiting requested review from VachaShah

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

backport 2.x Backport to 2.x branch Cluster Manager skip-changelog stalled Issues that have stalled

Projects

Cluster Manager Project Board
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jaideep-m @andrross