Skip to content

Optimization in String Terms Aggregation query for Large Bucket Counts#18732

Merged
rishabhmaurya merged 15 commits intoopensearch-project:mainfrom
vinaykpud:string-term-agg-opt
Sep 30, 2025
Merged

Optimization in String Terms Aggregation query for Large Bucket Counts#18732
rishabhmaurya merged 15 commits intoopensearch-project:mainfrom
vinaykpud:string-term-agg-opt

Conversation

@vinaykpud
Copy link
Copy Markdown
Contributor

@vinaykpud vinaykpud commented Jul 11, 2025 •
edited
Loading

Description

If the number of requested top-N buckets exceeds or close to the maximum bucket ordinal, making the use of a PriorityQueue for top-N selection inefficient or redundant. So we made following modifications:

  1. use quickselect for topN if the requested size is greater than the 20% of the total buckets.
  2. If the requested size is greater than the bucket size then return all the bucket.

Benchmark test results here :

#18704 (comment)

Related Issues

Resolves #18704
Related #18650

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added bug Something isn't working Search:Performance labels Jul 11, 2025
@vinaykpud vinaykpud force-pushed the string-term-agg-opt branch 4 times, most recently from fa96268 to 0cf5b78 Compare July 11, 2025 18:22
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 11, 2025

❌ Gradle check result for b25271f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 14, 2025

❌ Gradle check result for 242faae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@vinaykpud vinaykpud force-pushed the string-term-agg-opt branch from 242faae to 482a37e Compare July 14, 2025 22:34
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 14, 2025

❌ Gradle check result for 482a37e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@vinaykpud vinaykpud force-pushed the string-term-agg-opt branch from 482a37e to 81211c1 Compare July 14, 2025 23:46
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 15, 2025

❌ Gradle check result for 81211c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Jul 15, 2025
Open
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 15, 2025

❌ Gradle check result for a81608e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@vinaykpud vinaykpud closed this Jul 15, 2025
@vinaykpud vinaykpud reopened this Jul 15, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 15, 2025

❌ Gradle check result for a81608e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@vinaykpud vinaykpud marked this pull request as ready for review July 22, 2025 20:20
@vinaykpud
fixed bug in the condition
89f9b58 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@vinaykpud
Updated logic in topN selection depending on request size
8cd0b46 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@vinaykpud
use priority queue method for significant terms
01da1ee 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@vinaykpud
Added some comments
fc00b7e 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@vinaykpud
updated partiallyBuiltBucketComparator null check logic
7ec9e39 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@vinaykpud
Added tests and updated GlobalOrdinalsStringTermsAggregator
b455fc7 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@vinaykpud
Fixed spotless checks
a63bd29 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 26, 2025

❌ Gradle check result for b455fc7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@vinaykpud
Fixed issues in changelog
b6d4f94 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 26, 2025

✅ Gradle check result for b6d4f94: SUCCESS

@rishabhmaurya
Copy link
Copy Markdown
Contributor

rishabhmaurya commented Sep 30, 2025 •
edited
Loading

I'm guessing we are reusing the same logic we used for numeric terms #18702 i.e.

when size >= bucketsInOrd : return all
bucketsInOrd> size && bucketsInOrd <= 5*size : quick select
else i.e. bucketsInOrd> size &&   bucketsInOrd > 5*size : pq

LGTM

@vinaykpud
Copy link
Copy Markdown
Contributor Author

vinaykpud commented Sep 30, 2025

Correct, decision logic is same.

@vinaykpud
Merge branch 'main' into string-term-agg-opt
e3abf3e 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 30, 2025

❌ Gradle check result for e3abf3e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@vinaykpud vinaykpud closed this Sep 30, 2025
@vinaykpud vinaykpud reopened this Sep 30, 2025
@vinaykpud
Merge branch 'main' into string-term-agg-opt
f42714f 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 30, 2025

❕ Gradle check result for f42714f: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Sep 30, 2025
Open
@vinaykpud
Merge branch 'main' into string-term-agg-opt
33a1899 
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 30, 2025

✅ Gradle check result for 33a1899: SUCCESS

@rishabhmaurya rishabhmaurya merged commit 2817029 into opensearch-project:main Sep 30, 2025
33 checks passed
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Oct 9, 2025
Open
@BrewTestBot BrewTestBot mentioned this pull request Oct 15, 2025
Merged
peteralfonsi pushed a commit to peteralfonsi/OpenSearch that referenced this pull request Oct 15, 2025
@vinaykpud @rishabhmaurya
Optimization in String Terms Aggregation query for Large Bucket Counts (
72229f2 
opensearch-project#18732)

* Optimize String terms agg

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* Updated the algorithm selection logic and cleanup

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* Updated the algorithm selection logic and cleanup

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* Updated bucket sorting at shard level for keyorder

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* fixed bug in the condition

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* Updated logic in topN selection depending on request size

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* use priority queue method for significant terms

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* Added some comments

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* updated partiallyBuiltBucketComparator null check logic

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* Added tests and updated GlobalOrdinalsStringTermsAggregator

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* Fixed spotless checks

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

* Fixed issues in changelog

Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>

---------

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
Signed-off-by: Vinay Krishna Pudyodu <vinkrish.neo@gmail.com>
Co-authored-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Oct 22, 2025
Open
@tkykenmt tkykenmt mentioned this pull request Jan 10, 2026
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jainankitk jainankitk jainankitk left review comments

@rishabhmaurya rishabhmaurya rishabhmaurya approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@cwperks cwperks Awaiting requested review from cwperks cwperks is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah

@peternied peternied Awaiting requested review from peternied

Assignees

No one assigned

Labels

bug Something isn't working Search:Performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Performance] Optimize String terms agg

3 participants

@vinaykpud @rishabhmaurya @jainankitk