Skip to content

[ES|QL] Introduce CstToAstConverter in query parsing#225987

Merged
vadimkibana merged 7 commits intoelastic:mainfrom
vadimkibana:esql/parser-3
Jul 8, 2025
Merged

[ES|QL] Introduce CstToAstConverter in query parsing#225987
vadimkibana merged 7 commits intoelastic:mainfrom
vadimkibana:esql/parser-3

Conversation

@vadimkibana
Copy link
Copy Markdown
Contributor

@vadimkibana vadimkibana commented Jul 1, 2025 •
edited
Loading

Summary

Partially addresses #222505

  • Stops using ANTLR's listener API through ESQLAstBuilderListener (ESQLAstBuilderListener is completely removed) for ES|QL query parsing. Now all ANTLR CST (Concrete Syntax Tree) to AST (Abstract Syntax Tree) conversion is done by traversing the CST directly.
    • Consolidates most of the CST-to-AST conversion logic in the CstToAstConverter class.
    • In the future all conversion logic will be moved to the CstToAstConverter class.
  • This change now allows us to parse nested sub-queries, like:
    • EXPLAIN [ EXPLAIN [ FROM index ] ]
    • In the future: FROM index | WHERE foo IN (FROM bar | KEEP baz)
  • Improves FORK command parsing, now FORK command does not need to handle special cases of sub-query parsing, sub-query parsing "just works".
  • Fixes SHOW INFO command parsing.
  • Adds parsing unit tests for commands, which did no have them.

Checklist

  • Documentation was added for features that require explanation or tutorials

@vadimkibana vadimkibana changed the title add cst to ast converter and implement commands [ES|QL] Introduce CstToAstConverter in query parsing Jul 1, 2025
@vadimkibana vadimkibana added review v9.2.0 release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Feature:ES|QL ES|QL related features in Kibana Team:ESQL ES|QL related features in Kibana t// labels Jul 7, 2025
@vadimkibana vadimkibana marked this pull request as ready for review July 7, 2025 12:12
@vadimkibana vadimkibana requested a review from a team as a code owner July 7, 2025 12:12
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 7, 2025

Pinging @elastic/kibana-esql (Team:ESQL)

stratoula
stratoula approved these changes Jul 8, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@stratoula stratoula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, I just have a question why we didnt remove the entire factories folder

@vadimkibana
Copy link
Copy Markdown
Contributor Author

vadimkibana commented Jul 8, 2025

I just have a question why we didnt remove the entire factories folder

@stratoula in the first PR I wanted to introduce CstToAstConverter class, but now already working on consolidating all remaining code inside the CstToAstConverter.

@vadimkibana vadimkibana enabled auto-merge (squash) July 8, 2025 13:06
@vadimkibana vadimkibana merged commit 3ed9765 into elastic:main Jul 8, 2025
12 checks passed
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jul 8, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 7854 7831 -23

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/esql-ast 504 505 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 9.8MB 9.8MB +1.3KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
@kbn/esql-ast 34 33 -1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
kbnUiSharedDeps-srcJs 3.8MB 3.8MB +1.5KB
Unknown metric groups

API count

id before after diff
@kbn/esql-ast 595 600 +5

References to deprecated APIs

id before after diff
@kbn/esql-validation-autocomplete 11 8 -3

History

@drewdaemon
Copy link
Copy Markdown
Contributor

drewdaemon commented Jul 14, 2025

Nice! I like it!

kertal pushed a commit to kertal/kibana that referenced this pull request Jul 25, 2025
@vadimkibana @kertal
[ES|QL] Introduce CstToAstConverter in query parsing (elastic#225987)
22d2ae6 
## Summary

Partially addresses elastic#222505

- Stops using ANTLR's listener API through `ESQLAstBuilderListener`
(`ESQLAstBuilderListener` is completely removed) for ES|QL query
parsing. Now all ANTLR CST (Concrete Syntax Tree) to AST (Abstract
Syntax Tree) conversion is done by traversing the CST directly.
- Consolidates most of the CST-to-AST conversion logic in the
`CstToAstConverter` class.
- In the future all conversion logic will be moved to the
`CstToAstConverter` class.
- This change now allows us to parse nested sub-queries, like:
  - `EXPLAIN [ EXPLAIN [ FROM index ] ]`
  - In the future: `FROM index | WHERE foo IN (FROM bar | KEEP baz)`
- Improves `FORK` command parsing, now `FORK` command does not need to
handle special cases of sub-query parsing, sub-query parsing "just
works".
- Fixes `SHOW INFO` command parsing.
- Adds parsing unit tests for commands, which did no have them.

### Checklist

- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@stratoula stratoula stratoula approved these changes

Assignees

No one assigned

Labels

backport:skip This PR does not require backporting Feature:ES|QL ES|QL related features in Kibana release_note:skip Skip the PR/issue when compiling release notes review Team:ESQL ES|QL related features in Kibana t// v9.2.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vadimkibana @elasticmachine @drewdaemon @stratoula