Skip to content

[Enhancement]Merge group fields for aggregate if having dependent group fields#4703

Merged
LantaoJin merged 3 commits intoopensearch-project:mainfrom
qianheng-aws:4681
Oct 31, 2025
Merged

[Enhancement]Merge group fields for aggregate if having dependent group fields#4703
LantaoJin merged 3 commits intoopensearch-project:mainfrom
qianheng-aws:4681

Conversation

@qianheng-aws
Copy link
Copy Markdown
Collaborator

@qianheng-aws qianheng-aws commented Oct 30, 2025

Description

Merge group fields for aggregate if having dependent group fields. e.g. it will transform

... | eval a1 = a - 1, a2 = f(a), a3 = 1 | stats count() by a, a1, a2, a3

to

...| stats count() by a | eval a1 = a - 1, a2 = f(a), a3 = 1

which will both simplify aggregation complexity and also avoid agg-script push down.

Currently, we only support single base group fields and all other group fields must have dependency on the base one.

Related Issues

Resolves #4681

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…up fields

Signed-off-by: Heng Qian <qianheng@amazon.com>
Signed-off-by: Heng Qian <qianheng@amazon.com>
Signed-off-by: Heng Qian <qianheng@amazon.com>
Comment on lines +57 to +62
} else {
throw new AssertionError(
String.format(
"The length of rels should be %s but got %s",
this.operands.size(), call.rels.length));
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious when will there be more than 2 rels. With the given rule config, I assume it should match only the given pattern.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. Just defensive check as other rules.

@LantaoJin LantaoJin merged commit 5517c1e into opensearch-project:main Oct 31, 2025
33 checks passed
@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4703-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5517c1ea7cd11df8fffa1642c77b3dccc51c5e1e
# Push it to GitHub
git push --set-upstream origin backport/backport-4703-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4703-to-2.19-dev.

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4703-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5517c1ea7cd11df8fffa1642c77b3dccc51c5e1e
# Push it to GitHub
git push --set-upstream origin backport/backport-4703-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4703-to-2.19-dev.

qianheng-aws added a commit to qianheng-aws/sql that referenced this pull request Oct 31, 2025
…ensearch-project#4703)

* [Enhancement]Merge group fields for aggregate if having dependent group fields

Signed-off-by: Heng Qian <qianheng@amazon.com>

* fix CI

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix windows UT

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>

(cherry picked from commit 5517c1e)
Signed-off-by: Heng Qian <qianheng@amazon.com>
@LantaoJin LantaoJin added the backport-manually Filed a PR to backport manually. label Oct 31, 2025
LantaoJin pushed a commit that referenced this pull request Oct 31, 2025
…ent group fields #4703 (#4711)

* Merge group fields for aggregate if having dependent group fields (#4703)

* [Enhancement]Merge group fields for aggregate if having dependent group fields

Signed-off-by: Heng Qian <qianheng@amazon.com>

* fix CI

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix windows UT

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>

(cherry picked from commit 5517c1e)
Signed-off-by: Heng Qian <qianheng@amazon.com>

* fix compiling

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
expani pushed a commit to vinaykpud/sql that referenced this pull request Nov 4, 2025
…ensearch-project#4703)

* [Enhancement]Merge group fields for aggregate if having dependent group fields

Signed-off-by: Heng Qian <qianheng@amazon.com>

* fix CI

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix windows UT

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@qianheng-aws qianheng-aws deleted the 4681 branch March 26, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev backport-failed backport-manually Filed a PR to backport manually. calcite calcite migration releated clickbench enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Enhancement on agg with dependent bucket fields

3 participants