Field collapsing supports search_after by gaobinlong · Pull Request #19261 · opensearch-project/OpenSearch

gaobinlong · 2025-09-09T03:01:44Z

Description

Why field collapsing doesn't support pagination by search_after? The reason is that if we collapse and sort on different field, and use search_after to fetch next page, the same collapsed key may show again in the next page, see the example blow:

Now we collapse on session_id and sort by timestamp ascending, the first page would be:

s1 - /page1 - 2025-09-07T10:00:00
s2 - /home - 2025-09-07T10:02:00

, then paging with search_after, the second page would be:

s2 - /contact - 2025-09-07T10:07:00
s1 - /page2 - 2025-09-07T10:05:00

, the above behavior would break the field collapsing, but we could imagine that if we collapse and sort on same filed, search_after will work.

So this PR adds the search_after support for field collapsing, but limit the usage that collapse field and sort field must be the same and secondary sort is not supported.

Related Issues

Resolves #3725

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

github-actions · 2025-09-09T03:19:20Z

❌ Gradle check result for 9dc634d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

reta · 2025-09-11T10:44:16Z

Hi @reta @andrross , could you help to review this PR, thanks!

Apologies for the delay, @gaobinlong , I will take a look today / tomorrow, thank you

server/src/main/java/org/apache/lucene/search/grouping/CollapsingTopDocsCollector.java

server/src/main/java/org/opensearch/search/collapse/CollapseContext.java

server/src/main/java/org/opensearch/search/query/TopDocsCollectorContext.java

reta · 2025-09-13T17:02:34Z

Thanks a lot @gaobinlong , a few minor comments but LGTM otherwise, apologies for the delay

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

github-actions · 2025-09-16T05:30:57Z

❌ Gradle check result for ea9fd6b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

github-actions · 2025-09-16T06:09:55Z

❌ Gradle check result for fbf7a77: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-09-16T06:57:45Z

❌ Gradle check result for fbf7a77: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

github-actions · 2025-09-16T08:06:14Z

❌ Gradle check result for 62767e6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-09-16T12:06:30Z

❌ Gradle check result for 62767e6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-09-17T01:30:54Z

✅ Gradle check result for 62767e6: SUCCESS

codecov · 2025-09-17T01:31:15Z

Codecov Report

❌ Patch coverage is 77.55102% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.88%. Comparing base (a3ac746) to head (62767e6).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...rg/opensearch/search/collapse/CollapseContext.java	20.00%	2 Missing and 2 partials ⚠️
...main/java/org/opensearch/search/SearchService.java	0.00%	3 Missing ⚠️
...ne/search/grouping/CollapsingTopDocsCollector.java	93.75%	0 Missing and 2 partials ⚠️
...ensearch/search/query/TopDocsCollectorContext.java	77.77%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19261      +/-   ##
============================================
- Coverage     72.89%   72.88%   -0.01%     
- Complexity    69763    69765       +2     
============================================
  Files          5667     5667              
  Lines        320547   320590      +43     
  Branches      46348    46357       +9     
============================================
+ Hits         233648   233654       +6     
- Misses        68005    68038      +33     
- Partials      18894    18898       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

* Field collapsing supports search_after Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Modify change log Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Avoid changing public APIs Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Revert change for the existing method Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Format change log Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Fix some issues Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Remove duplicate code Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Fix format issue Signed-off-by: Binlong Gao <gbinlong@amazon.com> --------- Signed-off-by: Binlong Gao <gbinlong@amazon.com>

* Field collapsing supports search_after Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Modify change log Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Avoid changing public APIs Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Revert change for the existing method Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Format change log Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Fix some issues Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Remove duplicate code Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Fix format issue Signed-off-by: Binlong Gao <gbinlong@amazon.com> --------- Signed-off-by: Binlong Gao <gbinlong@amazon.com> Signed-off-by: Ankit Jain <jainankitk@apache.org>

* Field collapsing supports search_after Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Modify change log Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Avoid changing public APIs Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Revert change for the existing method Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Format change log Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Fix some issues Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Remove duplicate code Signed-off-by: Binlong Gao <gbinlong@amazon.com> * Fix format issue Signed-off-by: Binlong Gao <gbinlong@amazon.com> --------- Signed-off-by: Binlong Gao <gbinlong@amazon.com>

Field collapsing supports search_after

80bfc4a

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

gaobinlong requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners September 9, 2025 03:01

github-actions bot added enhancement Enhancement or improvement to existing feature or request feature New feature or request Indexing & Search labels Sep 9, 2025

gaobinlong added 2 commits September 9, 2025 11:02

Modify change log

d6ad5dd

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

Merge remote-tracking branch 'upstream/main' into searchAfter

9dc634d

gaobinlong mentioned this pull request Sep 9, 2025

[DOC] Field collapsing supports search_after opensearch-project/documentation-website#10896

Closed

4 tasks

gaobinlong added 2 commits September 9, 2025 11:29

Avoid changing public APIs

3e5ecd8

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

Revert change for the existing method

2fabc67

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

reta reviewed Sep 13, 2025

View reviewed changes

server/src/main/java/org/apache/lucene/search/grouping/CollapsingTopDocsCollector.java Outdated Show resolved Hide resolved

reta reviewed Sep 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/search/collapse/CollapseContext.java Outdated Show resolved Hide resolved

reta reviewed Sep 13, 2025

View reviewed changes

server/src/main/java/org/opensearch/search/query/TopDocsCollectorContext.java Outdated Show resolved Hide resolved

gaobinlong added 2 commits September 16, 2025 10:16

Merge remote-tracking branch 'upstream/main' into searchAfter

994644d

Fix some issues

ea9fd6b

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

Remove duplicate code

fbf7a77

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

Fix format issue

62767e6

Signed-off-by: Binlong Gao <gbinlong@amazon.com>

opensearch-ci-bot mentioned this pull request Sep 15, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SecureReactorNetty4HttpServerTransportTests #17486

Closed

reta approved these changes Sep 16, 2025

View reviewed changes

gaobinlong merged commit b856451 into opensearch-project:main Sep 17, 2025
33 of 37 checks passed

opensearch-ci-bot mentioned this pull request Sep 15, 2025

[AUTOCUT] Gradle Check Flaky Test Report for StreamStringTermsAggregatorTests #18956

Closed

BrewTestBot mentioned this pull request Oct 15, 2025

opensearch 3.3.0 Homebrew/homebrew-core#249627

Merged

penghuo mentioned this pull request Nov 12, 2025

Re-enable dedup pushdown opensearch-project/sql#4790

Closed

8 tasks

This was referenced Jan 10, 2026

docs: add field collapsing search_after report for v3.3.0 tkykenmt/opensearch-feature-explorer#1515

Merged

[feature] Field Collapsing tkykenmt/opensearch-feature-explorer#1394

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Field collapsing supports search_after#19261

Field collapsing supports search_after#19261
gaobinlong merged 10 commits intoopensearch-project:mainfrom
gaobinlong:searchAfter

gaobinlong commented Sep 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 9, 2025

Uh oh!

reta commented Sep 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reta commented Sep 13, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 17, 2025

Uh oh!

codecov bot commented Sep 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gaobinlong commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Sep 9, 2025

Uh oh!

reta commented Sep 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reta commented Sep 13, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 17, 2025

Uh oh!

codecov bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gaobinlong commented Sep 9, 2025 •

edited

Loading

codecov bot commented Sep 17, 2025 •

edited

Loading