Push down QUERY_SIZE_LIMIT by qianheng-aws · Pull Request #3880 · opensearch-project/sql

qianheng-aws · 2025-07-15T06:55:04Z

Description

Try to push down QUERY_SIZE_LIMIT if it's a simple plan with only scan.

This PR implements this by only pushing down QUERY_SIZE_LIMIT into the OpenSearchRequestBuilder of the final plan if it only contains a single scan operator.

This PR also fixes a bug in LIMIT push down by correct the cost computing logic. This bug will prevent multiple limit push down in some cases, see CalciteExplainIT::testMultipleLimitExplain.

Related Issues

Resolves #3879

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <qianheng@amazon.com>

penghuo · 2025-07-16T02:33:27Z

core/src/main/java/org/opensearch/sql/calcite/utils/CalciteToolsHelper.java

    }
    try {
-      return new OpenSearchDriver().connect("jdbc:calcite:", info, null, typeFactory);
+      return new OpenSearchDriver(querySizeLimit).connect("jdbc:calcite:", info, null, typeFactory);


Could we append Limit operator on each query instead of OpenSearchDriver constructor.

That's a workable options as well, I put it in the alternative solution. #3879 (comment).

It has side effect of changing the original plan if the append limit operator can not be push down.

OPTION1: Push down QUERY_SIZE_LIMIT to the final single scan

PROS: Won't change the plan, and the optimization process is efficient and straight forward

CONS: Only improve on the restricted case of the single scan

OPTION2: Append LIMIT operator on the original plan

PROS: Has improvement on more cases than single scan. e.g Project-Scan since it has SortProjectTransposeRule to swap Limit before the Project.

CONS: Will change the final plan if Limit operator cannot be push down

Which option do we prefer? @penghuo @LantaoJin

If we implement eval push down in the future, OPTION2 won't have its PROS since we can support pushing down all kinds of Project and left only a single Scan

Option 1 is more like a specific optimization for the setting query_size. I prefer to option 2 with a new LogicalSort such LogicalQueryLimit. Anyway, I am ok to both Option 1 and 2.

+1 on option2,

CONS: Will change the final plan if Limit operator cannot be push down

Could you elaborate on this? Are you suggesting that adding a LIMIT clause would change the result of the EXPLAIN plan? If that’s the concern, I’m okay with it. We explicitly enforce a querySizeLimit for every query anyway.

Latest impl is better, scanWithLimit is clean to me.

Are you suggesting that adding a LIMIT clause would change the result of the EXPLAIN plan?

Yes.

Changing the plan will have suspicion of changing customers' intension although we do have restriction on the final results.

And I was wondering why Hive or Spark doesn't do similar optimization of appending limit operator, it's said (by LLM) that the current implementation of counting row count on the final iteration has other advantages including:

Plan reuse for cases like pagination. Although we don't have such feature for PPL, but maybe needed in the future.

Better memory management. It currently doesn't applies for us since the whole process happens in one coordinator. But it will make sense once we change to distribution execution.

Keep the plan semantically equivalent to user's SQL.

So for long term consideration, keeping the plan unchanged makes more sense and is a standard practice.

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin

Note, the current implementation is option 1. I am also open to this option so I will approve it. @penghuo please merge it if the current implementation works for you too.

…QuerySizeLimit # Conflicts: # opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java

Signed-off-by: Heng Qian <qianheng@amazon.com>

opensearch-trigger-bot · 2025-07-17T18:06:27Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-3880-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0bded2aa4d9c80e07929fdc91bac1f74d1e18f7c
# Push it to GitHub
git push --set-upstream origin backport/backport-3880-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-3880-to-2.19-dev.

LantaoJin · 2025-07-18T04:10:20Z

@qianheng-aws please backport to 2.19-dev.

* Push down QUERY_SIZE_LIMIT Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix compiling Signed-off-by: Heng Qian <qianheng@amazon.com> * Use QUERY_SIZE_LIMIT in osIndex settings directly Signed-off-by: Heng Qian <qianheng@amazon.com> * Revert code Signed-off-by: Heng Qian <qianheng@amazon.com> * SpotlessApply after resolving conflict Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> (cherry picked from commit 0bded2a)

* Push down QUERY_SIZE_LIMIT Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix compiling Signed-off-by: Heng Qian <qianheng@amazon.com> * Use QUERY_SIZE_LIMIT in osIndex settings directly Signed-off-by: Heng Qian <qianheng@amazon.com> * Revert code Signed-off-by: Heng Qian <qianheng@amazon.com> * SpotlessApply after resolving conflict Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> (cherry picked from commit 0bded2a) Signed-off-by: Heng Qian <qianheng@amazon.com>

* Push down QUERY_SIZE_LIMIT * Fix compiling * Use QUERY_SIZE_LIMIT in osIndex settings directly * Revert code * SpotlessApply after resolving conflict --------- (cherry picked from commit 0bded2a) Signed-off-by: Heng Qian <qianheng@amazon.com>

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add LogicalSystemLimit Signed-off-by: Heng Qian <qianheng@amazon.com> * Revert part of #3880 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT after merging main Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com>

* Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add LogicalSystemLimit Signed-off-by: Heng Qian <qianheng@amazon.com> * Revert part of opensearch-project#3880 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT after merging main Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> (cherry picked from commit 5316c0a)

* Append limit operator for QUEERY_SIZE_LIMIT * Add LogicalSystemLimit * Revert part of #3880 * Fix IT after merging main --------- (cherry picked from commit 5316c0a) Signed-off-by: Heng Qian <qianheng@amazon.com>

* Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add LogicalSystemLimit Signed-off-by: Heng Qian <qianheng@amazon.com> * Revert part of opensearch-project#3880 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT after merging main Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com>

…cite) (#3970) * Add support for space-separated fields in addition to comma-separated Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Byte number should treated as Long in doc values (#3928) Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding table as alias of the fields command Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix create PIT permissions issue (#3921) Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Convert like function call to wildcard query for Calcite filter pushdown (#3915) * Convert like function call to wildcard query for Calcite filter pushdown Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix V2 expression like function bug and match its behavior in Calcite Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix like default escape in Calcite Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix tests Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix spotless check Signed-off-by: Songkan Tang <songkant@amazon.com> * Address comments Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix SQL IT correctness Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove test log Signed-off-by: Songkan Tang <songkant@amazon.com> * Minor improve one CalciteLikeQueryIT Signed-off-by: Songkan Tang <songkant@amazon.com> --------- Signed-off-by: Songkan Tang <songkant@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Update commons-lang exclude rule to exclude it everywhere (#3932) * Update commons-lang exclude rule to exclude it everywhere Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Undo removal in core Signed-off-by: Simeon Widdis <sawiddis@amazon.com> --------- Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding wildcard support to fields command in Calcite Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding widlcard support to non-Calcite engine and updating documentation with new features Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing formatting issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support function argument coercion with Calcite (#3914) * Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Implement basic argument type coercion at RelNode level Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Conform type checkers with their definition in documentation - string as an input is removed if it is not in the document - string as an input is kept if it is in the document, even if it can be implicitly cast - use PPLOperandTypes as much as possible Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Implement type widening for comparator functions - Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators - Implement widenArguments method in CoercionUtils to find widest compatible type - Apply type widening to comparator functions before applying type casting - Add detailed JavaDoc to explain coercion methods Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update error messages of datetime functions with invalid args Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify datetime-string compare logic with implict coercion Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor resolve with coercion Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Move down argument cast for reduce function Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Merge comparators and their IP variants so that coercion works for IP comparison - when not merging, ip comparing will also pass the type checker of Calcite's comparators Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor ip comparator to comparator Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert "Refactor ip comparator to comparator" This reverts commit c539056. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert "Merge comparators and their IP variants so that coercion works for IP comparison" This reverts commit bd9f3bb. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Rule out ip from built-in comparator via its type checker Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Restrict CompareIP's parameter type Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test argument coercion explain Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix error msg in CalcitePPLFunctionTypeTest Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add missing command in index.rst (#3943) Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Append limit operator for QUEERY_SIZE_LIMIT (#3940) * Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add LogicalSystemLimit Signed-off-by: Heng Qian <qianheng@amazon.com> * Revert part of #3880 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT after merging main Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Performing code cleaning and fixing tests Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Performing code maintenance and adding more test cases Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Doing some code cleaning and maintenance Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing code and implementation logic Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add issue template specific for PPL commands and queries (#3962) * Add issue template specific for PPL commands and queries Signed-off-by: Anas Alkouz <aalkouz@amazon.com> * Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas Signed-off-by: Anas Alkouz <aalkouz@amazon.com> --------- Signed-off-by: Anas Alkouz <aalkouz@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Increase the precision of sum return type (#3974) Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disable a failed PPL query fallback to v2 by default (#3952) * Disable a failed PPL query fallback to v2 by default Signed-off-by: Lantao Jin <ltjin@amazon.com> * Workaround the permissionIT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Update the maven snapshot publish endpoint and credential (#3806) Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add release notes for 3.2.0 (#3985) Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing documentation Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Taking care of comments left by Tomo Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding full wildcard support functionality Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Increment version to 3.2.0-SNAPSHOT (#3819) Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support `reverse` command with Calcite (#3867) * Implement reverse Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse integ tests and unit tests Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse documentation Signed-off-by: Selina Song <selsong@amazon.com> * Modify reverse test and documentation Signed-off-by: Selina Song <selsong@amazon.com> * Fix limit pushdown bug when reverse comes before head * Revert "Fix limit pushdown bug when reverse comes before head" This reverts commit 087c936. Signed-off-by: Selina Song <selsong@amazon.com> * Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR. Signed-off-by: Selina Song <selsong@amazon.com> * Fix reverse tests: update logical plans, format with Spotless - Updated expected logical plans and Spark SQL in reverse tests - Applied Spotless to fix formatting Signed-off-by: Selina Song <selsong@amazon.com> * Fix OS version in build 3.1.0 Signed-off-by: Selina Song <selsong@amazon.com> * Add note on limitation to rst Signed-off-by: Selina Song <selsong@amazon.com> * Move explain IT to correct file, add Anonymizer test Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse to index.rst Signed-off-by: Selina Song <selsong@amazon.com> --------- Signed-off-by: Selina Song <selsong@amazon.com> Co-authored-by: Selina Song <selsong@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Pass JOIN_TIME_OUT value to keepalive (#3826) * Fix JOIN_TIME_OUT Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix style Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix log-rethrow Signed-off-by: Kai Huang <ahkcs@amazon.com> * Restructure Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix format Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove unused PIT Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove reflection, add hintConfig Signed-off-by: Kai Huang <ahkcs@amazon.com> * Added Unit test, restructured to use existing methods Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * add IT tests Signed-off-by: Kai Huang <ahkcs@amazon.com> * formatting Signed-off-by: Kai Huang <ahkcs@amazon.com> * Trim IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * format fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * trim IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * formatting Signed-off-by: Kai Huang <ahkcs@amazon.com> * Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * update .gitignore Signed-off-by: Kai Huang <ahkcs@amazon.com> * deletion Signed-off-by: Kai Huang <ahkcs@amazon.com> * restore main PIT Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Changed seenFields to Hashset instead of LinkedHashSet Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Creating a rule only for the fields/table commands to avoid any interference Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * This commit is in response of PR comments left by Tomo and Chen Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing Integration test failure Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding anonymizer tests, wildcard unit tests, etc Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disabling Calcite for enhance fields features Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disabling automatic de-deduplication when Calcite is disabled Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding cross-cluster IT test Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding a dedicated Cross-cluster IT test file for Calcite Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing formatting issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Improving widlcard logic and exception message Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Addressing comments left by Tomo regarding wildcard logic implementation Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add missing udfs in v3 (#3957) * add math udfs Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix decimal bug Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * make general udf adapter Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add math IT Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add rst Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix error Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * change signum IT Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add javadoc Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> --------- Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * fix snapshot uploading (#4006) * fix snapshot uploading Signed-off-by: Kai Huang <ahkcs@amazon.com> * Add comment Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix DOUBLE to STRING cast rendering zero values in scientific notation (#3982) * Fix casting double 0.0 to string Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix float to string casting precision lost with custom FormatNumberFunction This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER with a custom FormatNumberFunction implementation. The new implementation converts the number to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming 6.199999809265137. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify the implementation of fp number to string cast Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update implementation of NumberToStringFunction Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Cast decimal with NUMBER_TO_STRING function Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test cast decimal Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Eliminate reliance on assert in Calcite for integration test (#4016) * Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Check script.disable_max_compilations_rate before setting context-specific compilations rate Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: remove some methods in tests to upper level to reduce duplication Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Replace plugin-level setting strings with private test-specific ones Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Prevent aggregation push down when it has inner filter (#4002) * Prevent aggregation push down when it has inner filter Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT & Remove log Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix 4009 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix span on negative timestamp (#4017) * Fix span on negative timestamp Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix span on negative timestamp Signed-off-by: Heng Qian <qianheng@amazon.com> * typo Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Skip script encoding when run explain with 'extended' (#3930) * No need to decode script when run explain command Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comment Signed-off-by: Lantao Jin <ltjin@amazon.com> * Do not encoding when explain format is 'extended' Signed-off-by: Lantao Jin <ltjin@amazon.com> * Rename the thread local var Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT after merge main Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Implement type checking for aggregation functions with Calcite (#4024) * Remove getTypeChecker from FunctionImp interface Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor registerExternalFunction to registerExternalOperator Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Do not register GEOIP function if got incompatible client Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Create scaffold for type checking of aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add type checkers for aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test type checking for aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Allow equal expression as a function argument (#4001) * Remove named function arg from functions other than table functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test eval if function with equal as condition Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Push down IP comparison as range query with Calcite (#3959) * Add reverse op for compare ip to support pushdown Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Pushdown ip comparison Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor CompareIpFunction to use SqlKind directly Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify the overriding of reverse() for IP comparators Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * eval sum, avg implementation (#3986) Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix PPL eval command string concatenation with + operator (#4020) * eval command support Signed-off-by: Kai Huang <ahkcs@amazon.com> * improvment Signed-off-by: Kai Huang <ahkcs@amazon.com> * Refactor Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * Add IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove redundant tests Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support script push down on text field (#4010) * Support script push down on text field Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add UT for struct type push down Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Enhance sort command in PPL (#3934) * enhance sort command Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add integ tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update documentation Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing test Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update default and tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update analyzer test Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update reverse sort direction Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update docs Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add javadoc Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update integ tests for query size limit change Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add explainit for desc and type cast Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add tests for desc Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * make count optional Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add cross cluster tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * normalize count in AST node Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * default null count to 0 Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update logicalsort default constructor Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> --------- Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add example for String concat in eval.rst (#4075) * Add example for String concat in eval.rst Signed-off-by: Kai Huang <ahkcs@amazon.com> * mention calcite enabling Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support pushdown dedup with Calcite (#3972) * Support pushdown dedup with Calcite Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * Address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix flaky test Signed-off-by: Lantao Jin <ltjin@amazon.com> * Address comment Signed-off-by: Lantao Jin <ltjin@amazon.com> * delete useless codes Signed-off-by: Lantao Jin <ltjin@amazon.com> * Add more ITs Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix CI failure because of plan having changed (#4077) Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> --------- Signed-off-by: Aaron Alvarez <aaarone@amazon.com> Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Songkan Tang <songkant@amazon.com> Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Anas Alkouz <aalkouz@amazon.com> Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Selina Song <selsong@amazon.com> Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> Signed-off-by: Aaron Alvarez <900908alvarezaaron@gmail.com> Co-authored-by: Aaron Alvarez <aaarone@amazon.com> Co-authored-by: Lantao Jin <ltjin@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com> Co-authored-by: Songkan Tang <songkant@amazon.com> Co-authored-by: Simeon Widdis <sawiddis@gmail.com> Co-authored-by: Yuanchun Shen <yuanchu@amazon.com> Co-authored-by: Peng Huo <penghuo@gmail.com> Co-authored-by: qianheng <qianheng@amazon.com> Co-authored-by: Anas Alkouz <aalkouz@amazon.com> Co-authored-by: Zelin Hao <zelinhao@amazon.com> Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com> Co-authored-by: opensearch-ci <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: Selina Song <selinasong6@gmail.com> Co-authored-by: Selina Song <selsong@amazon.com> Co-authored-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Xinyu Hao <75524174+ishaoxy@users.noreply.github.com> Co-authored-by: ritvibhatt <53196324+ritvibhatt@users.noreply.github.com>

…cite) (opensearch-project#3970) * Add support for space-separated fields in addition to comma-separated Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Byte number should treated as Long in doc values (opensearch-project#3928) Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding table as alias of the fields command Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix create PIT permissions issue (opensearch-project#3921) Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Convert like function call to wildcard query for Calcite filter pushdown (opensearch-project#3915) * Convert like function call to wildcard query for Calcite filter pushdown Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix V2 expression like function bug and match its behavior in Calcite Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix like default escape in Calcite Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix tests Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix spotless check Signed-off-by: Songkan Tang <songkant@amazon.com> * Address comments Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix SQL IT correctness Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove test log Signed-off-by: Songkan Tang <songkant@amazon.com> * Minor improve one CalciteLikeQueryIT Signed-off-by: Songkan Tang <songkant@amazon.com> --------- Signed-off-by: Songkan Tang <songkant@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Update commons-lang exclude rule to exclude it everywhere (opensearch-project#3932) * Update commons-lang exclude rule to exclude it everywhere Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Undo removal in core Signed-off-by: Simeon Widdis <sawiddis@amazon.com> --------- Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding wildcard support to fields command in Calcite Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding widlcard support to non-Calcite engine and updating documentation with new features Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing formatting issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support function argument coercion with Calcite (opensearch-project#3914) * Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Implement basic argument type coercion at RelNode level Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Conform type checkers with their definition in documentation - string as an input is removed if it is not in the document - string as an input is kept if it is in the document, even if it can be implicitly cast - use PPLOperandTypes as much as possible Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Implement type widening for comparator functions - Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators - Implement widenArguments method in CoercionUtils to find widest compatible type - Apply type widening to comparator functions before applying type casting - Add detailed JavaDoc to explain coercion methods Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update error messages of datetime functions with invalid args Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify datetime-string compare logic with implict coercion Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor resolve with coercion Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Move down argument cast for reduce function Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Merge comparators and their IP variants so that coercion works for IP comparison - when not merging, ip comparing will also pass the type checker of Calcite's comparators Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor ip comparator to comparator Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert "Refactor ip comparator to comparator" This reverts commit c539056. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert "Merge comparators and their IP variants so that coercion works for IP comparison" This reverts commit bd9f3bb. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Rule out ip from built-in comparator via its type checker Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Restrict CompareIP's parameter type Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test argument coercion explain Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix error msg in CalcitePPLFunctionTypeTest Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add missing command in index.rst (opensearch-project#3943) Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Append limit operator for QUEERY_SIZE_LIMIT (opensearch-project#3940) * Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add LogicalSystemLimit Signed-off-by: Heng Qian <qianheng@amazon.com> * Revert part of opensearch-project#3880 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT after merging main Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Performing code cleaning and fixing tests Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Performing code maintenance and adding more test cases Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Doing some code cleaning and maintenance Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing code and implementation logic Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add issue template specific for PPL commands and queries (opensearch-project#3962) * Add issue template specific for PPL commands and queries Signed-off-by: Anas Alkouz <aalkouz@amazon.com> * Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas Signed-off-by: Anas Alkouz <aalkouz@amazon.com> --------- Signed-off-by: Anas Alkouz <aalkouz@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Increase the precision of sum return type (opensearch-project#3974) Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disable a failed PPL query fallback to v2 by default (opensearch-project#3952) * Disable a failed PPL query fallback to v2 by default Signed-off-by: Lantao Jin <ltjin@amazon.com> * Workaround the permissionIT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Update the maven snapshot publish endpoint and credential (opensearch-project#3806) Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add release notes for 3.2.0 (opensearch-project#3985) Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing documentation Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Taking care of comments left by Tomo Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding full wildcard support functionality Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Increment version to 3.2.0-SNAPSHOT (opensearch-project#3819) Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support `reverse` command with Calcite (opensearch-project#3867) * Implement reverse Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse integ tests and unit tests Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse documentation Signed-off-by: Selina Song <selsong@amazon.com> * Modify reverse test and documentation Signed-off-by: Selina Song <selsong@amazon.com> * Fix limit pushdown bug when reverse comes before head * Revert "Fix limit pushdown bug when reverse comes before head" This reverts commit 087c936. Signed-off-by: Selina Song <selsong@amazon.com> * Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR. Signed-off-by: Selina Song <selsong@amazon.com> * Fix reverse tests: update logical plans, format with Spotless - Updated expected logical plans and Spark SQL in reverse tests - Applied Spotless to fix formatting Signed-off-by: Selina Song <selsong@amazon.com> * Fix OS version in build 3.1.0 Signed-off-by: Selina Song <selsong@amazon.com> * Add note on limitation to rst Signed-off-by: Selina Song <selsong@amazon.com> * Move explain IT to correct file, add Anonymizer test Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse to index.rst Signed-off-by: Selina Song <selsong@amazon.com> --------- Signed-off-by: Selina Song <selsong@amazon.com> Co-authored-by: Selina Song <selsong@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Pass JOIN_TIME_OUT value to keepalive (opensearch-project#3826) * Fix JOIN_TIME_OUT Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix style Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix log-rethrow Signed-off-by: Kai Huang <ahkcs@amazon.com> * Restructure Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix format Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove unused PIT Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove reflection, add hintConfig Signed-off-by: Kai Huang <ahkcs@amazon.com> * Added Unit test, restructured to use existing methods Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * add IT tests Signed-off-by: Kai Huang <ahkcs@amazon.com> * formatting Signed-off-by: Kai Huang <ahkcs@amazon.com> * Trim IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * format fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * trim IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * formatting Signed-off-by: Kai Huang <ahkcs@amazon.com> * Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * update .gitignore Signed-off-by: Kai Huang <ahkcs@amazon.com> * deletion Signed-off-by: Kai Huang <ahkcs@amazon.com> * restore main PIT Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Changed seenFields to Hashset instead of LinkedHashSet Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Creating a rule only for the fields/table commands to avoid any interference Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * This commit is in response of PR comments left by Tomo and Chen Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing Integration test failure Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding anonymizer tests, wildcard unit tests, etc Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disabling Calcite for enhance fields features Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disabling automatic de-deduplication when Calcite is disabled Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding cross-cluster IT test Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding a dedicated Cross-cluster IT test file for Calcite Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing formatting issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Improving widlcard logic and exception message Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Addressing comments left by Tomo regarding wildcard logic implementation Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add missing udfs in v3 (opensearch-project#3957) * add math udfs Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix decimal bug Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * make general udf adapter Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add math IT Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add rst Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix error Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * change signum IT Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add javadoc Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> --------- Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * fix snapshot uploading (opensearch-project#4006) * fix snapshot uploading Signed-off-by: Kai Huang <ahkcs@amazon.com> * Add comment Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix DOUBLE to STRING cast rendering zero values in scientific notation (opensearch-project#3982) * Fix casting double 0.0 to string Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix float to string casting precision lost with custom FormatNumberFunction This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER with a custom FormatNumberFunction implementation. The new implementation converts the number to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming 6.199999809265137. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify the implementation of fp number to string cast Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update implementation of NumberToStringFunction Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Cast decimal with NUMBER_TO_STRING function Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test cast decimal Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Eliminate reliance on assert in Calcite for integration test (opensearch-project#4016) * Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Check script.disable_max_compilations_rate before setting context-specific compilations rate Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: remove some methods in tests to upper level to reduce duplication Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Replace plugin-level setting strings with private test-specific ones Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Prevent aggregation push down when it has inner filter (opensearch-project#4002) * Prevent aggregation push down when it has inner filter Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT & Remove log Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix 4009 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix span on negative timestamp (opensearch-project#4017) * Fix span on negative timestamp Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix span on negative timestamp Signed-off-by: Heng Qian <qianheng@amazon.com> * typo Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Skip script encoding when run explain with 'extended' (opensearch-project#3930) * No need to decode script when run explain command Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comment Signed-off-by: Lantao Jin <ltjin@amazon.com> * Do not encoding when explain format is 'extended' Signed-off-by: Lantao Jin <ltjin@amazon.com> * Rename the thread local var Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT after merge main Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Implement type checking for aggregation functions with Calcite (opensearch-project#4024) * Remove getTypeChecker from FunctionImp interface Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor registerExternalFunction to registerExternalOperator Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Do not register GEOIP function if got incompatible client Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Create scaffold for type checking of aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add type checkers for aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test type checking for aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Allow equal expression as a function argument (opensearch-project#4001) * Remove named function arg from functions other than table functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test eval if function with equal as condition Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Push down IP comparison as range query with Calcite (opensearch-project#3959) * Add reverse op for compare ip to support pushdown Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Pushdown ip comparison Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor CompareIpFunction to use SqlKind directly Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify the overriding of reverse() for IP comparators Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * eval sum, avg implementation (opensearch-project#3986) Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix PPL eval command string concatenation with + operator (opensearch-project#4020) * eval command support Signed-off-by: Kai Huang <ahkcs@amazon.com> * improvment Signed-off-by: Kai Huang <ahkcs@amazon.com> * Refactor Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * Add IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove redundant tests Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support script push down on text field (opensearch-project#4010) * Support script push down on text field Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add UT for struct type push down Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Enhance sort command in PPL (opensearch-project#3934) * enhance sort command Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add integ tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update documentation Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing test Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update default and tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update analyzer test Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update reverse sort direction Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update docs Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add javadoc Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update integ tests for query size limit change Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add explainit for desc and type cast Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add tests for desc Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * make count optional Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add cross cluster tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * normalize count in AST node Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * default null count to 0 Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update logicalsort default constructor Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> --------- Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add example for String concat in eval.rst (opensearch-project#4075) * Add example for String concat in eval.rst Signed-off-by: Kai Huang <ahkcs@amazon.com> * mention calcite enabling Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support pushdown dedup with Calcite (opensearch-project#3972) * Support pushdown dedup with Calcite Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * Address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix flaky test Signed-off-by: Lantao Jin <ltjin@amazon.com> * Address comment Signed-off-by: Lantao Jin <ltjin@amazon.com> * delete useless codes Signed-off-by: Lantao Jin <ltjin@amazon.com> * Add more ITs Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix CI failure because of plan having changed (opensearch-project#4077) Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> --------- Signed-off-by: Aaron Alvarez <aaarone@amazon.com> Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Songkan Tang <songkant@amazon.com> Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Anas Alkouz <aalkouz@amazon.com> Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Selina Song <selsong@amazon.com> Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> Signed-off-by: Aaron Alvarez <900908alvarezaaron@gmail.com> Co-authored-by: Aaron Alvarez <aaarone@amazon.com> Co-authored-by: Lantao Jin <ltjin@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com> Co-authored-by: Songkan Tang <songkant@amazon.com> Co-authored-by: Simeon Widdis <sawiddis@gmail.com> Co-authored-by: Yuanchun Shen <yuanchu@amazon.com> Co-authored-by: Peng Huo <penghuo@gmail.com> Co-authored-by: qianheng <qianheng@amazon.com> Co-authored-by: Anas Alkouz <aalkouz@amazon.com> Co-authored-by: Zelin Hao <zelinhao@amazon.com> Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com> Co-authored-by: opensearch-ci <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: Selina Song <selinasong6@gmail.com> Co-authored-by: Selina Song <selsong@amazon.com> Co-authored-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Xinyu Hao <75524174+ishaoxy@users.noreply.github.com> Co-authored-by: ritvibhatt <53196324+ritvibhatt@users.noreply.github.com> (cherry picked from commit f16f1c0)

…ection Features (Cal… (#4102) * `fields` Command Enhancement - Advanced Field Selection Features (Calcite) (#3970) * Add support for space-separated fields in addition to comma-separated Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Byte number should treated as Long in doc values (#3928) Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding table as alias of the fields command Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix create PIT permissions issue (#3921) Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Convert like function call to wildcard query for Calcite filter pushdown (#3915) * Convert like function call to wildcard query for Calcite filter pushdown Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix V2 expression like function bug and match its behavior in Calcite Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix like default escape in Calcite Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix tests Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix spotless check Signed-off-by: Songkan Tang <songkant@amazon.com> * Address comments Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix SQL IT correctness Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove test log Signed-off-by: Songkan Tang <songkant@amazon.com> * Minor improve one CalciteLikeQueryIT Signed-off-by: Songkan Tang <songkant@amazon.com> --------- Signed-off-by: Songkan Tang <songkant@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Update commons-lang exclude rule to exclude it everywhere (#3932) * Update commons-lang exclude rule to exclude it everywhere Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Undo removal in core Signed-off-by: Simeon Widdis <sawiddis@amazon.com> --------- Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding wildcard support to fields command in Calcite Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding widlcard support to non-Calcite engine and updating documentation with new features Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing formatting issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support function argument coercion with Calcite (#3914) * Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Implement basic argument type coercion at RelNode level Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Conform type checkers with their definition in documentation - string as an input is removed if it is not in the document - string as an input is kept if it is in the document, even if it can be implicitly cast - use PPLOperandTypes as much as possible Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Implement type widening for comparator functions - Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators - Implement widenArguments method in CoercionUtils to find widest compatible type - Apply type widening to comparator functions before applying type casting - Add detailed JavaDoc to explain coercion methods Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update error messages of datetime functions with invalid args Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify datetime-string compare logic with implict coercion Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor resolve with coercion Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Move down argument cast for reduce function Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Merge comparators and their IP variants so that coercion works for IP comparison - when not merging, ip comparing will also pass the type checker of Calcite's comparators Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor ip comparator to comparator Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert "Refactor ip comparator to comparator" This reverts commit c539056. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert "Merge comparators and their IP variants so that coercion works for IP comparison" This reverts commit bd9f3bb. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Rule out ip from built-in comparator via its type checker Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Restrict CompareIP's parameter type Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test argument coercion explain Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix error msg in CalcitePPLFunctionTypeTest Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add missing command in index.rst (#3943) Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Append limit operator for QUEERY_SIZE_LIMIT (#3940) * Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add LogicalSystemLimit Signed-off-by: Heng Qian <qianheng@amazon.com> * Revert part of #3880 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT after merging main Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Performing code cleaning and fixing tests Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Performing code maintenance and adding more test cases Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Doing some code cleaning and maintenance Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing code and implementation logic Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add issue template specific for PPL commands and queries (#3962) * Add issue template specific for PPL commands and queries Signed-off-by: Anas Alkouz <aalkouz@amazon.com> * Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas Signed-off-by: Anas Alkouz <aalkouz@amazon.com> --------- Signed-off-by: Anas Alkouz <aalkouz@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Increase the precision of sum return type (#3974) Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disable a failed PPL query fallback to v2 by default (#3952) * Disable a failed PPL query fallback to v2 by default Signed-off-by: Lantao Jin <ltjin@amazon.com> * Workaround the permissionIT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Update the maven snapshot publish endpoint and credential (#3806) Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add release notes for 3.2.0 (#3985) Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing documentation Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Taking care of comments left by Tomo Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding full wildcard support functionality Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Increment version to 3.2.0-SNAPSHOT (#3819) Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support `reverse` command with Calcite (#3867) * Implement reverse Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse integ tests and unit tests Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse documentation Signed-off-by: Selina Song <selsong@amazon.com> * Modify reverse test and documentation Signed-off-by: Selina Song <selsong@amazon.com> * Fix limit pushdown bug when reverse comes before head * Revert "Fix limit pushdown bug when reverse comes before head" This reverts commit 087c936. Signed-off-by: Selina Song <selsong@amazon.com> * Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR. Signed-off-by: Selina Song <selsong@amazon.com> * Fix reverse tests: update logical plans, format with Spotless - Updated expected logical plans and Spark SQL in reverse tests - Applied Spotless to fix formatting Signed-off-by: Selina Song <selsong@amazon.com> * Fix OS version in build 3.1.0 Signed-off-by: Selina Song <selsong@amazon.com> * Add note on limitation to rst Signed-off-by: Selina Song <selsong@amazon.com> * Move explain IT to correct file, add Anonymizer test Signed-off-by: Selina Song <selsong@amazon.com> * Add reverse to index.rst Signed-off-by: Selina Song <selsong@amazon.com> --------- Signed-off-by: Selina Song <selsong@amazon.com> Co-authored-by: Selina Song <selsong@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Pass JOIN_TIME_OUT value to keepalive (#3826) * Fix JOIN_TIME_OUT Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix style Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix log-rethrow Signed-off-by: Kai Huang <ahkcs@amazon.com> * Restructure Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix format Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove unused PIT Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove reflection, add hintConfig Signed-off-by: Kai Huang <ahkcs@amazon.com> * Added Unit test, restructured to use existing methods Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * add IT tests Signed-off-by: Kai Huang <ahkcs@amazon.com> * formatting Signed-off-by: Kai Huang <ahkcs@amazon.com> * Trim IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * format fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * trim IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * formatting Signed-off-by: Kai Huang <ahkcs@amazon.com> * Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * update .gitignore Signed-off-by: Kai Huang <ahkcs@amazon.com> * deletion Signed-off-by: Kai Huang <ahkcs@amazon.com> * restore main PIT Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Changed seenFields to Hashset instead of LinkedHashSet Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Creating a rule only for the fields/table commands to avoid any interference Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * This commit is in response of PR comments left by Tomo and Chen Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing Integration test failure Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding anonymizer tests, wildcard unit tests, etc Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disabling Calcite for enhance fields features Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Disabling automatic de-deduplication when Calcite is disabled Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding cross-cluster IT test Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding a dedicated Cross-cluster IT test file for Calcite Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing formatting issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Improving widlcard logic and exception message Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Addressing comments left by Tomo regarding wildcard logic implementation Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add missing udfs in v3 (#3957) * add math udfs Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix decimal bug Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * make general udf adapter Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add math IT Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add rst Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * fix error Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * change signum IT Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> * add javadoc Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> --------- Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * fix snapshot uploading (#4006) * fix snapshot uploading Signed-off-by: Kai Huang <ahkcs@amazon.com> * Add comment Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix DOUBLE to STRING cast rendering zero values in scientific notation (#3982) * Fix casting double 0.0 to string Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix float to string casting precision lost with custom FormatNumberFunction This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER with a custom FormatNumberFunction implementation. The new implementation converts the number to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming 6.199999809265137. Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify the implementation of fp number to string cast Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update implementation of NumberToStringFunction Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Cast decimal with NUMBER_TO_STRING function Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test cast decimal Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Eliminate reliance on assert in Calcite for integration test (#4016) * Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Check script.disable_max_compilations_rate before setting context-specific compilations rate Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor: remove some methods in tests to upper level to reduce duplication Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Replace plugin-level setting strings with private test-specific ones Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Prevent aggregation push down when it has inner filter (#4002) * Prevent aggregation push down when it has inner filter Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT & Remove log Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix 4009 Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix span on negative timestamp (#4017) * Fix span on negative timestamp Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix span on negative timestamp Signed-off-by: Heng Qian <qianheng@amazon.com> * typo Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Skip script encoding when run explain with 'extended' (#3930) * No need to decode script when run explain command Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comment Signed-off-by: Lantao Jin <ltjin@amazon.com> * Do not encoding when explain format is 'extended' Signed-off-by: Lantao Jin <ltjin@amazon.com> * Rename the thread local var Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT after merge main Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Implement type checking for aggregation functions with Calcite (#4024) * Remove getTypeChecker from FunctionImp interface Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor registerExternalFunction to registerExternalOperator Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Do not register GEOIP function if got incompatible client Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Create scaffold for type checking of aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add type checkers for aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test type checking for aggregation functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Allow equal expression as a function argument (#4001) * Remove named function arg from functions other than table functions Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test eval if function with equal as condition Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Push down IP comparison as range query with Calcite (#3959) * Add reverse op for compare ip to support pushdown Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Pushdown ip comparison Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Refactor CompareIpFunction to use SqlKind directly Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify the overriding of reverse() for IP comparators Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * eval sum, avg implementation (#3986) Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix PPL eval command string concatenation with + operator (#4020) * eval command support Signed-off-by: Kai Huang <ahkcs@amazon.com> * improvment Signed-off-by: Kai Huang <ahkcs@amazon.com> * Refactor Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix CI Signed-off-by: Kai Huang <ahkcs@amazon.com> * fixes Signed-off-by: Kai Huang <ahkcs@amazon.com> * fix Signed-off-by: Kai Huang <ahkcs@amazon.com> * Add IT Signed-off-by: Kai Huang <ahkcs@amazon.com> * remove redundant tests Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support script push down on text field (#4010) * Support script push down on text field Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> * Add UT for struct type push down Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Enhance sort command in PPL (#3934) * enhance sort command Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add integ tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update documentation Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing test Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update default and tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update analyzer test Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update reverse sort direction Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update docs Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add javadoc Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix failing tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update integ tests for query size limit change Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add explainit for desc and type cast Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add tests for desc Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix formatting Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * make count optional Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * add cross cluster tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * normalize count in AST node Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * default null count to 0 Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * update logicalsort default constructor Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> --------- Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add example for String concat in eval.rst (#4075) * Add example for String concat in eval.rst Signed-off-by: Kai Huang <ahkcs@amazon.com> * mention calcite enabling Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Support pushdown dedup with Calcite (#3972) * Support pushdown dedup with Calcite Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix IT Signed-off-by: Lantao Jin <ltjin@amazon.com> * Address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix flaky test Signed-off-by: Lantao Jin <ltjin@amazon.com> * Address comment Signed-off-by: Lantao Jin <ltjin@amazon.com> * delete useless codes Signed-off-by: Lantao Jin <ltjin@amazon.com> * Add more ITs Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix CI failure because of plan having changed (#4077) Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Empty commit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> --------- Signed-off-by: Aaron Alvarez <aaarone@amazon.com> Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Songkan Tang <songkant@amazon.com> Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Anas Alkouz <aalkouz@amazon.com> Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Selina Song <selsong@amazon.com> Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> Signed-off-by: Aaron Alvarez <900908alvarezaaron@gmail.com> Co-authored-by: Aaron Alvarez <aaarone@amazon.com> Co-authored-by: Lantao Jin <ltjin@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com> Co-authored-by: Songkan Tang <songkant@amazon.com> Co-authored-by: Simeon Widdis <sawiddis@gmail.com> Co-authored-by: Yuanchun Shen <yuanchu@amazon.com> Co-authored-by: Peng Huo <penghuo@gmail.com> Co-authored-by: qianheng <qianheng@amazon.com> Co-authored-by: Anas Alkouz <aalkouz@amazon.com> Co-authored-by: Zelin Hao <zelinhao@amazon.com> Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com> Co-authored-by: opensearch-ci <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: Selina Song <selinasong6@gmail.com> Co-authored-by: Selina Song <selsong@amazon.com> Co-authored-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Xinyu Hao <75524174+ishaoxy@users.noreply.github.com> Co-authored-by: ritvibhatt <53196324+ritvibhatt@users.noreply.github.com> (cherry picked from commit f16f1c0) * Fixing compatibility issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing JDK compatibility issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * fixing java 11 compatibility issues Signed-off-by: Aaron Alvarez <aaarone@amazon.com> --------- Signed-off-by: Aaron Alvarez <aaarone@amazon.com> Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Songkan Tang <songkant@amazon.com> Signed-off-by: Simeon Widdis <sawiddis@amazon.com> Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: Peng Huo <penghuo@gmail.com> Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: Anas Alkouz <aalkouz@amazon.com> Signed-off-by: opensearch-ci <opensearch-infra@amazon.com> Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Selina Song <selsong@amazon.com> Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Signed-off-by: Xinyu Hao <haoxinyu@amazon.com> Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> Signed-off-by: Aaron Alvarez <900908alvarezaaron@gmail.com> Co-authored-by: Aaron Alvarez <aaarone@amazon.com> Co-authored-by: Lantao Jin <ltjin@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com> Co-authored-by: Songkan Tang <songkant@amazon.com> Co-authored-by: Simeon Widdis <sawiddis@gmail.com> Co-authored-by: Yuanchun Shen <yuanchu@amazon.com> Co-authored-by: Peng Huo <penghuo@gmail.com> Co-authored-by: qianheng <qianheng@amazon.com> Co-authored-by: Anas Alkouz <aalkouz@amazon.com> Co-authored-by: Zelin Hao <zelinhao@amazon.com> Co-authored-by: Sayali Gaikawad <gaiksaya@amazon.com> Co-authored-by: opensearch-ci <83309141+opensearch-ci-bot@users.noreply.github.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: Selina Song <selinasong6@gmail.com> Co-authored-by: Selina Song <selsong@amazon.com> Co-authored-by: Kai Huang <105710027+ahkcs@users.noreply.github.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Xinyu Hao <75524174+ishaoxy@users.noreply.github.com> Co-authored-by: ritvibhatt <53196324+ritvibhatt@users.noreply.github.com>

Push down QUERY_SIZE_LIMIT

4eea3c6

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az and ykmr1224 as code owners July 15, 2025 06:55

qianheng-aws added enhancement New feature or request calcite calcite migration releated pushdown pushdown related issues backport 2.19-dev labels Jul 15, 2025

Fix compiling

c8b80df

Signed-off-by: Heng Qian <qianheng@amazon.com>

penghuo reviewed Jul 16, 2025

View reviewed changes

qianheng-aws added 2 commits July 16, 2025 16:54

Use QUERY_SIZE_LIMIT in osIndex settings directly

96351b0

Signed-off-by: Heng Qian <qianheng@amazon.com>

Revert code

0fc83f2

Signed-off-by: Heng Qian <qianheng@amazon.com>

penghuo previously approved these changes Jul 16, 2025

View reviewed changes

LantaoJin previously approved these changes Jul 17, 2025

View reviewed changes

qianheng-aws added 2 commits July 17, 2025 14:19

Merge remote-tracking branch 'refs/remotes/origin/main' into PushDown…

5827d63

…QuerySizeLimit # Conflicts: # opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/AbstractCalciteIndexScan.java

SpotlessApply after resolving conflict

6f80257

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws dismissed stale reviews from LantaoJin and penghuo via 6f80257 July 17, 2025 06:27

LantaoJin approved these changes Jul 17, 2025

View reviewed changes

penghuo approved these changes Jul 17, 2025

View reviewed changes

penghuo merged commit 0bded2a into opensearch-project:main Jul 17, 2025
23 checks passed

opensearch-trigger-bot bot added the backport-failed label Jul 17, 2025

qianheng-aws mentioned this pull request Jul 30, 2025

Append limit operator for QUEERY_SIZE_LIMIT #3940

Merged

7 tasks

LantaoJin added the backport-manually Filed a PR to backport manually. label Jul 31, 2025

qianheng-aws added a commit to qianheng-aws/sql that referenced this pull request Jul 31, 2025

Revert part of opensearch-project#3880

7217650

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws mentioned this pull request Aug 1, 2025

[Backport 2.19-dev]Append limit operator for QUEERY_SIZE_LIMIT (#3940) #3966

Merged

7 tasks

opensearch-project deleted a comment from opensearch-trigger-bot bot Aug 13, 2025

qianheng-aws deleted the PushDownQuerySizeLimit branch March 26, 2026 10:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Push down QUERY_SIZE_LIMIT#3880

Push down QUERY_SIZE_LIMIT#3880
penghuo merged 6 commits intoopensearch-project:mainfrom
qianheng-aws:PushDownQuerySizeLimit

qianheng-aws commented Jul 15, 2025

Uh oh!

penghuo Jul 16, 2025

Uh oh!

qianheng-aws Jul 16, 2025

Uh oh!

qianheng-aws Jul 16, 2025

Uh oh!

LantaoJin Jul 16, 2025 •

edited

Loading

Uh oh!

penghuo Jul 16, 2025 •

edited

Loading

Uh oh!

qianheng-aws Jul 17, 2025

Uh oh!

LantaoJin left a comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Jul 17, 2025

Uh oh!

LantaoJin commented Jul 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

qianheng-aws commented Jul 15, 2025

Description

Related Issues

Check List

Uh oh!

penghuo Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penghuo Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Jul 17, 2025

Uh oh!

LantaoJin commented Jul 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LantaoJin Jul 16, 2025 •

edited

Loading

penghuo Jul 16, 2025 •

edited

Loading