Introduce `execution_hint` for Cardinality aggregation by asimmahmood1 · Pull Request #17312 · opensearch-project/OpenSearch

asimmahmood1 · 2025-02-10T19:35:57Z

Description

Introduce execution_hint for Cardinality aggregation - Revive PR #15764

This is intended as short term solution as a workaround to users that see large latency regression when upgrading to OS 2.x. The main risk here is ordinals take up more memory, i.e. depends on size of cardinality. So for very high cardinality, there is a risk of OOM at shard node. This is marked as power user.

I plan to follow up with #15269 which is more useful setting. This will allow the user select a higher threshold for using ordinals, e.g. a shard with large heap can use ordinals more often, but still fall back to the slower direct as needed.

Related Issues

Resolves #16837.

Check List

Functionality includes testing.
API changes companion pull request [n/a]
Public documentation issue/PR created,

Testing

Tested using big5 dataset, see #16837 (comment)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

opensearch-trigger-bot · 2025-02-20T22:07:28Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-17312-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e3a6ccadc942c64e83bd224031bc4d1c6ab14623
# Push it to GitHub
git push --set-upstream origin backport/backport-17312-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-17312-to-2.x.

…oject#17312) - backport to 2.19 --------- Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com> Signed-off-by: Asim Mahmood <asim.seng@gmail.com> Signed-off-by: Asim M <asim.seng@gmail.com> Co-authored-by: Siddharth Rayabharam <maitreya2954@gmail.com> Co-authored-by: Craig Perkins <craig5008@gmail.com> (cherry picked from commit e3a6cca)

…kport to 2.19 (#17420) * Introduce `execution_hint` for Cardinality aggregation (#17312) - backport to 2.19 --------- Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com> Signed-off-by: Asim Mahmood <asim.seng@gmail.com> Signed-off-by: Asim M <asim.seng@gmail.com> Co-authored-by: Siddharth Rayabharam <maitreya2954@gmail.com> Co-authored-by: Craig Perkins <craig5008@gmail.com> (cherry picked from commit e3a6cca) * Update changelog Signed-off-by: Asim Mahmood <asim.seng@gmail.com> --------- Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

…kport to 2.x (#17419) * Introduce `execution_hint` for Cardinality aggregation (#17312) --------- Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com> Signed-off-by: Asim Mahmood <asim.seng@gmail.com> Signed-off-by: Asim M <asim.seng@gmail.com> Co-authored-by: Siddharth Rayabharam <maitreya2954@gmail.com> Co-authored-by: Craig Perkins <craig5008@gmail.com> (cherry picked from commit e3a6cca) * Update changlog Signed-off-by: Asim Mahmood <asim.seng@gmail.com> * Update min version to 2_19_1 * since 2.19 version has been merged: #17420 Signed-off-by: Asim Mahmood <asim.seng@gmail.com> --------- Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

…execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

…execution hint ordinals (#579) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <asim.seng@gmail.com> * Update check to include version less than 6.0.0 to handle old ES versions Signed-off-by: Asim Mahmood <asim.seng@gmail.com> * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name Signed-off-by: Asim Mahmood <asim.seng@gmail.com> --------- Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

…execution hint ordinals (#579) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <asim.seng@gmail.com> * Update check to include version less than 6.0.0 to handle old ES versions Signed-off-by: Asim Mahmood <asim.seng@gmail.com> * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name Signed-off-by: Asim Mahmood <asim.seng@gmail.com> --------- Signed-off-by: Asim Mahmood <asim.seng@gmail.com> (cherry picked from commit 2bbfefc) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…execution hint ordinals (#579) (#601) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality * Update check to include version less than 6.0.0 to handle old ES versions * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name --------- (cherry picked from commit 2bbfefc) Signed-off-by: Asim Mahmood <asim.seng@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…execution hint ordinals (#579) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <asim.seng@gmail.com> * Update check to include version less than 6.0.0 to handle old ES versions Signed-off-by: Asim Mahmood <asim.seng@gmail.com> * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name Signed-off-by: Asim Mahmood <asim.seng@gmail.com> --------- Signed-off-by: Asim Mahmood <asim.seng@gmail.com> (cherry picked from commit 2bbfefc) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…execution hint ordinals (#579) (#614) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality * Update check to include version less than 6.0.0 to handle old ES versions * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name --------- (cherry picked from commit 2bbfefc) Signed-off-by: Asim Mahmood <asim.seng@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

maitreya2954 added 10 commits February 10, 2025 19:27

Introduce execution hint for Cardinality aggregation

6aa63ab

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

Cleanup ExecutionMode enum declarations

1e012ad

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

Remove unwanted lines

fc333c3

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

toString overrided for ExecutionHint

342108a

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

Testcases added for CardinalityAggregator execution hint

5f0b5a8

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

Test case for invalid execution hint added

c8964f3

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

Java tags added for ExecutionMode

92020b3

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

Test method names corrected

40ab27c

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

gradle format checks

3471f4d

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

Forbidden Apis fixed

746df72

Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>

asimmahmood1 requested review from Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dblock, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, sachinpkale, saratvemulapalli and shwetathareja as code owners February 10, 2025 19:35

asimmahmood1 mentioned this pull request Feb 17, 2025

[Feature Request] Add a cluster setting for memory threshold to pick OrdinalsCollector in Cardinality aggregation #15269

Closed

sandeshkr419 approved these changes Feb 17, 2025

View reviewed changes

cwperks approved these changes Feb 18, 2025

View reviewed changes

asimmahmood1 requested a review from msfroh February 18, 2025 17:55

msfroh approved these changes Feb 20, 2025

View reviewed changes

msfroh merged commit e3a6cca into opensearch-project:main Feb 20, 2025
33 of 36 checks passed

This was referenced Feb 20, 2025

[AUTOCUT] Gradle Check Flaky Test Report for PendingTasksBlocksIT #17369

Open

[AUTOCUT] Gradle Check Flaky Test Report for IndexStatsIT #15836

Open

asimmahmood1 deleted the execution_hint branch February 20, 2025 17:52

mch2 added the backport 2.x Backport to 2.x branch label Feb 20, 2025

opensearch-trigger-bot bot added the backport-failed label Feb 20, 2025

asimmahmood1 mentioned this pull request Feb 21, 2025

Introduce execution_hint for Cardinality aggregation (#17312) - backport to 2.x #17419

Merged

3 tasks

asimmahmood1 mentioned this pull request Feb 21, 2025

Introduce execution_hint for Cardinality aggregation (#17312) - backport to 2.19 #17420

Merged

3 tasks

prudhvigodithi mentioned this pull request Feb 28, 2025

Search only replicas (scale to zero) with Reader/Writer Separation #17299

Merged

3 tasks

asimmahmood1 mentioned this pull request Mar 13, 2025

Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals opensearch-project/opensearch-benchmark-workloads#579

Merged

6 tasks

BrewTestBot mentioned this pull request May 6, 2025

opensearch 3.0.0 Homebrew/homebrew-core#222665

Merged

karenyrx mentioned this pull request Oct 30, 2025

Update TermsAggregation, CardinalityExecutionMode fields opensearch-project/opensearch-api-specification#982

Merged

This was referenced Jan 12, 2026

docs: add Aggregations report for v3.0.0 tkykenmt/opensearch-feature-explorer#1936

Merged

[v3.0.0] Aggregations tkykenmt/opensearch-feature-explorer#224

Closed

jainankitk mentioned this pull request Jan 23, 2026

Introduce execution hint for Cardinality aggregation #15764

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `execution_hint` for Cardinality aggregation#17312

Introduce `execution_hint` for Cardinality aggregation#17312
msfroh merged 23 commits intoopensearch-project:mainfrom
asimmahmood1:execution_hint

asimmahmood1 commented Feb 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Feb 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

asimmahmood1 commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Testing

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Feb 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

asimmahmood1 commented Feb 10, 2025 •

edited

Loading