Skip to content

Fix flaky testMaxScoreWithSortOnScoreAndCollapsingResults#19513

Merged
gaobinlong merged 2 commits intoopensearch-project:mainfrom
bowenlan-amzn:flaky-test
Oct 17, 2025
Merged

Fix flaky testMaxScoreWithSortOnScoreAndCollapsingResults#19513
gaobinlong merged 2 commits intoopensearch-project:mainfrom
bowenlan-amzn:flaky-test

Conversation

@bowenlan-amzn
Copy link
Copy Markdown
Member

@bowenlan-amzn bowenlan-amzn commented Oct 3, 2025

Description

Fixes a flaky test in QueryPhaseTests.testMaxScoreWithSortOnScoreAndCollapsingResults that fails intermittently with certain random seeds, like A86CC83E4E8823F2.

./gradlew ':server:test' --tests 'org.opensearch.search.query.QueryPhaseTests' \
    -Dtests.method='testMaxScoreWithSortOnScoreAndCollapsingResults' \
    -Dtests.seed=A86CC83E4E8823F2

Example failure: https://build.ci.opensearch.org/job/gradle-check/64947/testReport/junit/org.opensearch.search.query/QueryPhaseTests/testMaxScoreWithSortOnScoreAndCollapsingResults__p0_5_p1_org_opensearch_search_query_ConcurrentQueryPhaseSearcher_1f6ba47b_/

Test came from #19181

The test was making order-dependent assertions on collapsed search results:

assertThat(topDocs.collapseValues[0], equalTo(0L)); // user == 0
assertThat(topDocs.collapseValues[1], equalTo(1L)); // user == 1

However, the order of collapsed results is non-deterministic when documents have equal scores because:

  1. All test documents match the query foo:bar with identical scores
  2. Documents are indexed with alternating user values (0, 1, 0, 1, ...)
  3. RandomIndexWriter may reorder/merge documents during indexing based on the random seed
  4. With certain seeds (e.g., A86CC83E4E8823F2), documents with user=1 end up at lower doc IDs than user=0
  5. When collapsing on the user field with equal scores, the document encountered first (lower doc ID) is returned
  6. This causes the collapse values to be [1, 0] instead of the expected [0, 1]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Oct 3, 2025

❕ Gradle check result for 2d9b1d3: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@codecov
Copy link
Copy Markdown

codecov bot commented Oct 3, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.98%. Comparing base (ac6dfa1) to head (7a7055d).
⚠️ Report is 57 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #19513      +/-   ##
============================================
- Coverage     73.09%   72.98%   -0.11%     
+ Complexity    70553    70409     -144     
============================================
  Files          5716     5716              
  Lines        322926   322926              
  Branches      46770    46770              
============================================
- Hits         236032   235694     -338     
- Misses        67882    68221     +339     
+ Partials      19012    19011       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bowenlan-amzn
Copy link
Copy Markdown
Member Author

@gaobinlong please take a look when you got time

@gaobinlong
Copy link
Copy Markdown
Contributor

Thanks @bowenlan-amzn, by comparing with testCollapseQuerySearchResults(), I found that index sorting is missing in this method, so add an index sorting configuration to the IndexWriter is a better solution?

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

✅ Gradle check result for 7a7055d: SUCCESS

@gaobinlong gaobinlong merged commit 5df1f80 into opensearch-project:main Oct 17, 2025
33 checks passed
@bowenlan-amzn bowenlan-amzn deleted the flaky-test branch October 17, 2025 16:01
rgsriram pushed a commit to rgsriram/OpenSearch that referenced this pull request Oct 18, 2025
…-project#19513)

* fix flaky testMaxScoreWithSortOnScoreAndCollapsingResults

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

* fix flaky testMaxScoreWithSortOnScoreAndCollapsingResults

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

---------

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>
kh3ra pushed a commit to kh3ra/OpenSearch that referenced this pull request Oct 23, 2025
…-project#19513)

* fix flaky testMaxScoreWithSortOnScoreAndCollapsingResults

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

* fix flaky testMaxScoreWithSortOnScoreAndCollapsingResults

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

---------

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants