Add Tree Query Language and Engine by AntonyBlakey · Pull Request #753 · NomicFoundation/slang

AntonyBlakey · 2024-01-21T04:30:22Z

This pull request adds the CST Tree Query Language.

It is a long way from being optimised, and in particular it should not need to allocate memory whilst backtracking, which it currently does.

It is built with understandability as the top priority.

This also includes some other changes that are not quite drive-by e.g. I rationalised the runtime crate structure and file naming to make more sense given I was adding a structural sub-module.

changeset-bot · 2024-01-21T04:30:26Z

🦋 Changeset detected

Latest commit: 6dc5ef6

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@nomicfoundation/slang	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

crates/codegen/parser/runtime/src/templates/kinds.rs.jinja2

crates/codegen/parser/runtime/src/cursor.rs

crates/codegen/parser/runtime/src/kinds.rs

crates/codegen/parser/runtime/src/lib.rs

crates/solidity/outputs/npm/crate/src/generated/napi/napi_ast_selectors.rs

Cargo.toml

package-lock.json

crates/codegen/parser/runtime/src/query_model.rs

crates/codegen/parser/runtime/src/query_parser_tests.rs

crates/codegen/parser/runtime/src/query_engine.rs

crates/codegen/parser/runtime/src/query_engine_tests.rs

OmarTawfik

Left a few questions/suggestions.
Otherwise, this looks awesome! 🎉

crates/codegen/parser/runtime/src/kinds.rs

crates/codegen/parser/runtime/src/query/engine.rs

OmarTawfik

PR looks great. Accepted to unblock you, but there are a couple of remaining concerns I would suggest addressing before merging (phf, and the cursor/query public API). The others are just spelling/nits.

@AntonyBlakey

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @nomicfoundation/slang@0.14.0 ### Minor Changes - [#753](#753) [`b35c763`](b35c763) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Add tree query implementation as `Query::parse` and `Cursor::query` - [#755](#755) [`8c260fc`](8c260fc) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support parsing NatSpec comments - [#908](#908) [`ab3688b`](ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Changed the cst.NodeType in TS to use more descriptive string values rather than 0/1 integers - [#886](#886) [`0125717`](0125717) Thanks [@Xanewok](https://github.com/Xanewok)! - Add `TokenKind::is_trivia` - [#887](#887) [`dff1201`](dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Add support for constant function modifier removed in 0.5.0 - [#885](#885) [`a9bd8da`](a9bd8da) Thanks [@Xanewok](https://github.com/Xanewok)! - Flatten the trivia syntax nodes into sibling tokens - [#908](#908) [`ab3688b`](ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Add RuleNode/TokenNode::toJSON() in the TypeScript API ### Patch Changes - [#801](#801) [`ecbba49`](ecbba49) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - unreserve pragma keywords in all versions - [#869](#869) [`951b58d`](951b58d) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support dots in yul identifiers from `0.5.8` till `0.7.0` - [#890](#890) [`1ff8599`](1ff8599) Thanks [@Xanewok](https://github.com/Xanewok)! - Mark `override` as being a valid attribute only after 0.6.0 - [#800](#800) [`d1827ff`](d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support unicode characters in string literals up to `0.7.0` - [#797](#797) [`86f36d7`](86f36d7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix source locations for unicode characters in error reports - [#854](#854) [`4b8970b`](4b8970b) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - parse line breaks without newlines - [#844](#844) [`f62de9e`](f62de9e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing empty `/**/` comments - [#799](#799) [`303dda9`](303dda9) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prevent parsing multiple literals under `StringExpression` before `0.5.14` - [#847](#847) [`6b6f260`](6b6f260) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prioritize parsing `MultiLineComment` over `MultiLineNatSpecComment` - [#796](#796) [`59e1e53`](59e1e53) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `public` and `internal` to `UnnamedFunctionAttribute` till `0.5.0` - [#756](#756) [`e839817`](e839817) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing `payable` primary expressions - [#851](#851) [`67dfde8`](67dfde8) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix selection order of prefix/postfix AST fields - [#857](#857) [`f677d5e`](f677d5e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `FieldName` to `NodeLabel` - [#852](#852) [`ca79eca`](ca79eca) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow parsing `ColonEqual` as two separate tokens before `0.5.5` - [#889](#889) [`ce5050f`](ce5050f) Thanks [@Xanewok](https://github.com/Xanewok)! - Support `delete` as an expression rather than a statement - [#923](#923) [`bb30fc1`](bb30fc1) Thanks [@Xanewok](https://github.com/Xanewok)! - Support arbitrary ASCII escape sequences in string literals until 0.4.25 - [#887](#887) [`dff1201`](dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Support view and pure function modifiers only from 0.4.16 - [#800](#800) [`d1827ff`](d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `AsciiStringLiteral` to `StringLiteral` - [#838](#838) [`ad98d1c`](ad98d1c) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - upgrade to rust `1.76.0` - [#849](#849) [`5c42e0e`](5c42e0e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `override` and `virtual` to `ConstructorAttribute` - [#862](#862) [`5e37ea0`](5e37ea0) Thanks [@Xanewok](https://github.com/Xanewok)! - allow call options as a post-fix expression - [#786](#786) [`0bfa6b7`](0bfa6b7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support Yul label statements before `0.5.0` - [#839](#839) [`2d698eb`](2d698eb) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support string literals in version pragmas - [#891](#891) [`70c9d7d`](70c9d7d) Thanks [@Xanewok](https://github.com/Xanewok)! - Fix parsing `<NUMBER>.member` member access expression - [#842](#842) [`2069126`](2069126) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `private` to `UnnamedFunctionAttribute` till `0.5.0` - [#840](#840) [`7fb0d20`](7fb0d20) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow `var` in `TupleDeconstructionStatement` before `0.5.0` Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

@AntonyBlakey

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @nomicfoundation/slang@0.14.0 ### Minor Changes - [#753](NomicFoundation/slang#753) [`b35c763`](NomicFoundation/slang@b35c763) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Add tree query implementation as `Query::parse` and `Cursor::query` - [#755](NomicFoundation/slang#755) [`8c260fc`](NomicFoundation/slang@8c260fc) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support parsing NatSpec comments - [#908](NomicFoundation/slang#908) [`ab3688b`](NomicFoundation/slang@ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Changed the cst.NodeType in TS to use more descriptive string values rather than 0/1 integers - [#886](NomicFoundation/slang#886) [`0125717`](NomicFoundation/slang@0125717) Thanks [@Xanewok](https://github.com/Xanewok)! - Add `TokenKind::is_trivia` - [#887](NomicFoundation/slang#887) [`dff1201`](NomicFoundation/slang@dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Add support for constant function modifier removed in 0.5.0 - [#885](NomicFoundation/slang#885) [`a9bd8da`](NomicFoundation/slang@a9bd8da) Thanks [@Xanewok](https://github.com/Xanewok)! - Flatten the trivia syntax nodes into sibling tokens - [#908](NomicFoundation/slang#908) [`ab3688b`](NomicFoundation/slang@ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Add RuleNode/TokenNode::toJSON() in the TypeScript API ### Patch Changes - [#801](NomicFoundation/slang#801) [`ecbba49`](NomicFoundation/slang@ecbba49) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - unreserve pragma keywords in all versions - [#869](NomicFoundation/slang#869) [`951b58d`](NomicFoundation/slang@951b58d) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support dots in yul identifiers from `0.5.8` till `0.7.0` - [#890](NomicFoundation/slang#890) [`1ff8599`](NomicFoundation/slang@1ff8599) Thanks [@Xanewok](https://github.com/Xanewok)! - Mark `override` as being a valid attribute only after 0.6.0 - [#800](NomicFoundation/slang#800) [`d1827ff`](NomicFoundation/slang@d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support unicode characters in string literals up to `0.7.0` - [#797](NomicFoundation/slang#797) [`86f36d7`](NomicFoundation/slang@86f36d7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix source locations for unicode characters in error reports - [#854](NomicFoundation/slang#854) [`4b8970b`](NomicFoundation/slang@4b8970b) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - parse line breaks without newlines - [#844](NomicFoundation/slang#844) [`f62de9e`](NomicFoundation/slang@f62de9e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing empty `/**/` comments - [#799](NomicFoundation/slang#799) [`303dda9`](NomicFoundation/slang@303dda9) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prevent parsing multiple literals under `StringExpression` before `0.5.14` - [#847](NomicFoundation/slang#847) [`6b6f260`](NomicFoundation/slang@6b6f260) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prioritize parsing `MultiLineComment` over `MultiLineNatSpecComment` - [#796](NomicFoundation/slang#796) [`59e1e53`](NomicFoundation/slang@59e1e53) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `public` and `internal` to `UnnamedFunctionAttribute` till `0.5.0` - [#756](NomicFoundation/slang#756) [`e839817`](NomicFoundation/slang@e839817) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing `payable` primary expressions - [#851](NomicFoundation/slang#851) [`67dfde8`](NomicFoundation/slang@67dfde8) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix selection order of prefix/postfix AST fields - [#857](NomicFoundation/slang#857) [`f677d5e`](NomicFoundation/slang@f677d5e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `FieldName` to `NodeLabel` - [#852](NomicFoundation/slang#852) [`ca79eca`](NomicFoundation/slang@ca79eca) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow parsing `ColonEqual` as two separate tokens before `0.5.5` - [#889](NomicFoundation/slang#889) [`ce5050f`](NomicFoundation/slang@ce5050f) Thanks [@Xanewok](https://github.com/Xanewok)! - Support `delete` as an expression rather than a statement - [#923](NomicFoundation/slang#923) [`bb30fc1`](NomicFoundation/slang@bb30fc1) Thanks [@Xanewok](https://github.com/Xanewok)! - Support arbitrary ASCII escape sequences in string literals until 0.4.25 - [#887](NomicFoundation/slang#887) [`dff1201`](NomicFoundation/slang@dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Support view and pure function modifiers only from 0.4.16 - [#800](NomicFoundation/slang#800) [`d1827ff`](NomicFoundation/slang@d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `AsciiStringLiteral` to `StringLiteral` - [#838](NomicFoundation/slang#838) [`ad98d1c`](NomicFoundation/slang@ad98d1c) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - upgrade to rust `1.76.0` - [#849](NomicFoundation/slang#849) [`5c42e0e`](NomicFoundation/slang@5c42e0e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `override` and `virtual` to `ConstructorAttribute` - [#862](NomicFoundation/slang#862) [`5e37ea0`](NomicFoundation/slang@5e37ea0) Thanks [@Xanewok](https://github.com/Xanewok)! - allow call options as a post-fix expression - [#786](NomicFoundation/slang#786) [`0bfa6b7`](NomicFoundation/slang@0bfa6b7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support Yul label statements before `0.5.0` - [#839](NomicFoundation/slang#839) [`2d698eb`](NomicFoundation/slang@2d698eb) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support string literals in version pragmas - [#891](NomicFoundation/slang#891) [`70c9d7d`](NomicFoundation/slang@70c9d7d) Thanks [@Xanewok](https://github.com/Xanewok)! - Fix parsing `<NUMBER>.member` member access expression - [#842](NomicFoundation/slang#842) [`2069126`](NomicFoundation/slang@2069126) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `private` to `UnnamedFunctionAttribute` till `0.5.0` - [#840](NomicFoundation/slang#840) [`7fb0d20`](NomicFoundation/slang@7fb0d20) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow `var` in `TupleDeconstructionStatement` before `0.5.0` Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

AntonyBlakey requested a review from a team as a code owner January 21, 2024 04:30

AntonyBlakey self-assigned this Jan 21, 2024

Tree query language and engine

0bc7f81