Support format=yaml in Explain API by penghuo · Pull Request #4446 · opensearch-project/sql

penghuo · 2025-10-06T23:39:59Z

Description

Add the YAML explain format responses. It more human readable.
Refresh user documentation. Example 4 YAML format (experimental). Marked it as experimental, the reasons is YAML format is more for developer to debug and test. The response may change.

calcite:
  logical: |
    LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])
      LogicalProject(name=[$0], country=[$1], state=[$2], month=[$3], year=[$4], age=[$5])
        LogicalFilter(condition=[>($5, 30)])
          CalciteLogicalIndexScan(table=[[OpenSearch, state_country]])
  physical: |
    CalciteEnumerableIndexScan(table=[[OpenSearch, state_country]], PushDownContext=[[PROJECT->[name, country, state, month, year, age], FILTER->>($5, 30), LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":10000,"timeout":"1m","query":{"range":{"age":{"from":30,"to":null,"include_lower":false,"include_upper":true,"boost":1.0}}},"_source":{"includes":["name","country","state","month","year","age"],"excludes":[]}}, requestedTotalSize=10000, pageSize=null, startFrom=0)])

Related Issues

#4351

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch/src/main/java/org/opensearch/sql/opensearch/executor/OpenSearchExecutionEngine.java

Swiddis · 2025-10-07T18:28:48Z

opensearch/src/main/java/org/opensearch/sql/opensearch/executor/OpenSearchExecutionEngine.java

 import org.opensearch.sql.storage.TableScanOperator;

 /** OpenSearch execution engine implementation. */
 public class OpenSearchExecutionEngine implements ExecutionEngine {


suggestion (non-blocking): Should we move the explain functionality to a dedicated location?

The class is somewhat large as-is, and I don't like the idea of adding more complexity to OpenSearchExecutionEngine.

Good point!

I like your PR. add yaml format to minic that.

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo · 2025-10-10T15:12:19Z

docs/category.json

    "user/admin/settings.rst"
  ],
-  "ppl_cli": [
+  "bash_calcite": [


Test PPL with calcite

penghuo · 2025-10-10T15:19:17Z

docs/user/ppl/interfaces/endpoint.rst

+-----------------------------------

-The following PPL query demonstrated that where and stats command were pushed down to OpenSearch DSL aggregation query::
+Explain query::


Refresh doctest will correct response. Not breaking change.

penghuo · 2025-10-10T15:20:52Z

integ-test/src/test/java/org/opensearch/sql/ppl/ExplainIT.java

+    assertYamlEqualsIgnoreId(
        expected,
-        explainQueryToString(
+        explainQueryYaml(


Leverage Yaml response format.
Deprecated assertYamlEqualsJsonIgnoreId function. @ykmr1224 Please take a look

penghuo · 2025-10-10T15:23:11Z

protocol/src/main/java/org/opensearch/sql/protocol/response/format/Format.java

    builder.put(STANDARD.formatName, STANDARD);
    builder.put(EXTENDED.formatName, EXTENDED);
    builder.put(COST.formatName, COST);


These not response FORMAT. We should deprecated these in 4.0 and suggest user using explain statement

Signed-off-by: Peng Huo <penghuo@gmail.com>

opensearch-trigger-bot · 2025-10-20T20:02:52Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4446-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d6a9719f3c8d73f1b274842c5b83e7df7df5d9ca
# Push it to GitHub
git push --set-upstream origin backport/backport-4446-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4446-to-2.19-dev.

RyanL1997 · 2025-10-20T20:15:19Z

may need to manual backport it

--------- Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo · 2025-10-20T23:32:02Z

Manual backport #4609

--------- Signed-off-by: Peng Huo <penghuo@gmail.com>

* Support format=yaml in Explain API (#4446) --------- Signed-off-by: Peng Huo <penghuo@gmail.com> * Fix compile issue in main (#4608) Signed-off-by: Peng Huo <penghuo@gmail.com> * Update Signed-off-by: Peng Huo <penghuo@gmail.com> * update doc Signed-off-by: Peng Huo <penghuo@gmail.com> * Update Signed-off-by: Peng Huo <penghuo@gmail.com> --------- Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo added PPL Piped processing language enhancement New feature or request labels Oct 6, 2025

penghuo self-assigned this Oct 6, 2025

penghuo added the backport 2.19-dev label Oct 7, 2025

penghuo marked this pull request as ready for review October 7, 2025 18:02

Swiddis reviewed Oct 7, 2025

View reviewed changes

opensearch/src/main/java/org/opensearch/sql/opensearch/executor/OpenSearchExecutionEngine.java Outdated Show resolved Hide resolved

Swiddis reviewed Oct 7, 2025

View reviewed changes

Add Explain yaml format

9ffe777

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo commented Oct 10, 2025

View reviewed changes

docs/category.json

"user/admin/settings.rst"

],

"ppl_cli": [

"bash_calcite": [

Copy link
Copy Markdown

Collaborator Author

penghuo Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test PPL with calcite

penghuo commented Oct 10, 2025

View reviewed changes

penghuo added 11 commits October 10, 2025 08:23

Merge branch 'main' into issue4351

0b32657

Signed-off-by: Peng Huo <penghuo@gmail.com>

update

6b1de41

Signed-off-by: Peng Huo <penghuo@gmail.com>

Fix compile issue

071db7a

Signed-off-by: Peng Huo <penghuo@gmail.com>

Fix UT

0464207

Signed-off-by: Peng Huo <penghuo@gmail.com>

Fix UT

f66beb7

Signed-off-by: Peng Huo <penghuo@gmail.com>

Fix IT

2fa0d3f

Signed-off-by: Peng Huo <penghuo@gmail.com>

Update DocTest

c03facf

Signed-off-by: Peng Huo <penghuo@gmail.com>

Update

e8f0277

Signed-off-by: Peng Huo <penghuo@gmail.com>

Update

aa3b21f

Signed-off-by: Peng Huo <penghuo@gmail.com>

Update

454bf83

Signed-off-by: Peng Huo <penghuo@gmail.com>

Update

64bcd96

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo marked this pull request as draft October 11, 2025 00:45

Update

0e01e7b

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo marked this pull request as ready for review October 11, 2025 02:41

ykmr1224 approved these changes Oct 14, 2025

View reviewed changes

RyanL1997 approved these changes Oct 20, 2025

View reviewed changes

penghuo merged commit d6a9719 into opensearch-project:main Oct 20, 2025
33 checks passed

opensearch-trigger-bot bot added the backport-failed label Oct 20, 2025

penghuo added a commit to penghuo/os-sql that referenced this pull request Oct 20, 2025

Support format=yaml in Explain API (opensearch-project#4446)

e597942

--------- Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo mentioned this pull request Oct 20, 2025

[Backport 2.19-dev] Support format=yaml in Explain API (#4446) #4609

Merged

8 tasks

penghuo added a commit to penghuo/os-sql that referenced this pull request Oct 22, 2025

Support format=yaml in Explain API (opensearch-project#4446)

5efa478

--------- Signed-off-by: Peng Huo <penghuo@gmail.com>

LantaoJin added the backport-manually Filed a PR to backport manually. label Oct 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support format=yaml in Explain API#4446

Support format=yaml in Explain API#4446
penghuo merged 13 commits intoopensearch-project:mainfrom
penghuo:issue4351

penghuo commented Oct 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

Swiddis Oct 7, 2025

Uh oh!

penghuo Oct 10, 2025

Uh oh!

penghuo Oct 10, 2025

Uh oh!

penghuo Oct 10, 2025

Uh oh!

penghuo Oct 10, 2025 •

edited

Loading

Uh oh!

penghuo Oct 10, 2025

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Oct 20, 2025

Uh oh!

RyanL1997 commented Oct 20, 2025

Uh oh!

penghuo commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

penghuo commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Uh oh!

Swiddis Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

penghuo Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

penghuo Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

penghuo Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

penghuo Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penghuo Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Oct 20, 2025

Uh oh!

RyanL1997 commented Oct 20, 2025

Uh oh!

penghuo commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

penghuo commented Oct 6, 2025 •

edited

Loading

penghuo Oct 10, 2025 •

edited

Loading