Emit trivia as sibling tokens and don't generate a `RuleKind` for it by Xanewok · Pull Request #885 · NomicFoundation/slang

Xanewok · 2024-03-05T20:35:21Z

Closes #737

Functionally it's done but needs a rebase and a final polish. For now I'm using the existing PG infra until we clean it up (#638, slated for this sprint as well).

changeset-bot · 2024-03-05T20:35:25Z

🦋 Changeset detected

Latest commit: 341f04d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@nomicfoundation/slang	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Xanewok · 2024-03-05T20:36:26Z

Actually, it might be good enough for a first review pass 🤞

OmarTawfik · 2024-03-07T16:34:21Z

crates/codegen/grammar/src/constructor.rs

@@ -45,12 +45,12 @@ impl GrammarConstructorDslV2 for Grammar {

        let leading_trivia = Rc::new(NamedTriviaParser {
            name: "LeadingTrivia",


I suggest adding a change.set here about the API changes.

Building block for #862 and #885 Hopefully extracting this into a smaller PR can speed up its landing, as other refactors also touch the tree shape/v1 PG etc.

@AntonyBlakey

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @nomicfoundation/slang@0.14.0 ### Minor Changes - [#753](#753) [`b35c763`](b35c763) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Add tree query implementation as `Query::parse` and `Cursor::query` - [#755](#755) [`8c260fc`](8c260fc) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support parsing NatSpec comments - [#908](#908) [`ab3688b`](ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Changed the cst.NodeType in TS to use more descriptive string values rather than 0/1 integers - [#886](#886) [`0125717`](0125717) Thanks [@Xanewok](https://github.com/Xanewok)! - Add `TokenKind::is_trivia` - [#887](#887) [`dff1201`](dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Add support for constant function modifier removed in 0.5.0 - [#885](#885) [`a9bd8da`](a9bd8da) Thanks [@Xanewok](https://github.com/Xanewok)! - Flatten the trivia syntax nodes into sibling tokens - [#908](#908) [`ab3688b`](ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Add RuleNode/TokenNode::toJSON() in the TypeScript API ### Patch Changes - [#801](#801) [`ecbba49`](ecbba49) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - unreserve pragma keywords in all versions - [#869](#869) [`951b58d`](951b58d) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support dots in yul identifiers from `0.5.8` till `0.7.0` - [#890](#890) [`1ff8599`](1ff8599) Thanks [@Xanewok](https://github.com/Xanewok)! - Mark `override` as being a valid attribute only after 0.6.0 - [#800](#800) [`d1827ff`](d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support unicode characters in string literals up to `0.7.0` - [#797](#797) [`86f36d7`](86f36d7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix source locations for unicode characters in error reports - [#854](#854) [`4b8970b`](4b8970b) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - parse line breaks without newlines - [#844](#844) [`f62de9e`](f62de9e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing empty `/**/` comments - [#799](#799) [`303dda9`](303dda9) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prevent parsing multiple literals under `StringExpression` before `0.5.14` - [#847](#847) [`6b6f260`](6b6f260) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prioritize parsing `MultiLineComment` over `MultiLineNatSpecComment` - [#796](#796) [`59e1e53`](59e1e53) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `public` and `internal` to `UnnamedFunctionAttribute` till `0.5.0` - [#756](#756) [`e839817`](e839817) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing `payable` primary expressions - [#851](#851) [`67dfde8`](67dfde8) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix selection order of prefix/postfix AST fields - [#857](#857) [`f677d5e`](f677d5e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `FieldName` to `NodeLabel` - [#852](#852) [`ca79eca`](ca79eca) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow parsing `ColonEqual` as two separate tokens before `0.5.5` - [#889](#889) [`ce5050f`](ce5050f) Thanks [@Xanewok](https://github.com/Xanewok)! - Support `delete` as an expression rather than a statement - [#923](#923) [`bb30fc1`](bb30fc1) Thanks [@Xanewok](https://github.com/Xanewok)! - Support arbitrary ASCII escape sequences in string literals until 0.4.25 - [#887](#887) [`dff1201`](dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Support view and pure function modifiers only from 0.4.16 - [#800](#800) [`d1827ff`](d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `AsciiStringLiteral` to `StringLiteral` - [#838](#838) [`ad98d1c`](ad98d1c) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - upgrade to rust `1.76.0` - [#849](#849) [`5c42e0e`](5c42e0e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `override` and `virtual` to `ConstructorAttribute` - [#862](#862) [`5e37ea0`](5e37ea0) Thanks [@Xanewok](https://github.com/Xanewok)! - allow call options as a post-fix expression - [#786](#786) [`0bfa6b7`](0bfa6b7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support Yul label statements before `0.5.0` - [#839](#839) [`2d698eb`](2d698eb) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support string literals in version pragmas - [#891](#891) [`70c9d7d`](70c9d7d) Thanks [@Xanewok](https://github.com/Xanewok)! - Fix parsing `<NUMBER>.member` member access expression - [#842](#842) [`2069126`](2069126) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `private` to `UnnamedFunctionAttribute` till `0.5.0` - [#840](#840) [`7fb0d20`](7fb0d20) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow `var` in `TupleDeconstructionStatement` before `0.5.0` Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

@AntonyBlakey

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @nomicfoundation/slang@0.14.0 ### Minor Changes - [#753](NomicFoundation/slang#753) [`b35c763`](NomicFoundation/slang@b35c763) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Add tree query implementation as `Query::parse` and `Cursor::query` - [#755](NomicFoundation/slang#755) [`8c260fc`](NomicFoundation/slang@8c260fc) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support parsing NatSpec comments - [#908](NomicFoundation/slang#908) [`ab3688b`](NomicFoundation/slang@ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Changed the cst.NodeType in TS to use more descriptive string values rather than 0/1 integers - [#886](NomicFoundation/slang#886) [`0125717`](NomicFoundation/slang@0125717) Thanks [@Xanewok](https://github.com/Xanewok)! - Add `TokenKind::is_trivia` - [#887](NomicFoundation/slang#887) [`dff1201`](NomicFoundation/slang@dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Add support for constant function modifier removed in 0.5.0 - [#885](NomicFoundation/slang#885) [`a9bd8da`](NomicFoundation/slang@a9bd8da) Thanks [@Xanewok](https://github.com/Xanewok)! - Flatten the trivia syntax nodes into sibling tokens - [#908](NomicFoundation/slang#908) [`ab3688b`](NomicFoundation/slang@ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Add RuleNode/TokenNode::toJSON() in the TypeScript API ### Patch Changes - [#801](NomicFoundation/slang#801) [`ecbba49`](NomicFoundation/slang@ecbba49) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - unreserve pragma keywords in all versions - [#869](NomicFoundation/slang#869) [`951b58d`](NomicFoundation/slang@951b58d) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support dots in yul identifiers from `0.5.8` till `0.7.0` - [#890](NomicFoundation/slang#890) [`1ff8599`](NomicFoundation/slang@1ff8599) Thanks [@Xanewok](https://github.com/Xanewok)! - Mark `override` as being a valid attribute only after 0.6.0 - [#800](NomicFoundation/slang#800) [`d1827ff`](NomicFoundation/slang@d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support unicode characters in string literals up to `0.7.0` - [#797](NomicFoundation/slang#797) [`86f36d7`](NomicFoundation/slang@86f36d7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix source locations for unicode characters in error reports - [#854](NomicFoundation/slang#854) [`4b8970b`](NomicFoundation/slang@4b8970b) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - parse line breaks without newlines - [#844](NomicFoundation/slang#844) [`f62de9e`](NomicFoundation/slang@f62de9e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing empty `/**/` comments - [#799](NomicFoundation/slang#799) [`303dda9`](NomicFoundation/slang@303dda9) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prevent parsing multiple literals under `StringExpression` before `0.5.14` - [#847](NomicFoundation/slang#847) [`6b6f260`](NomicFoundation/slang@6b6f260) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prioritize parsing `MultiLineComment` over `MultiLineNatSpecComment` - [#796](NomicFoundation/slang#796) [`59e1e53`](NomicFoundation/slang@59e1e53) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `public` and `internal` to `UnnamedFunctionAttribute` till `0.5.0` - [#756](NomicFoundation/slang#756) [`e839817`](NomicFoundation/slang@e839817) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing `payable` primary expressions - [#851](NomicFoundation/slang#851) [`67dfde8`](NomicFoundation/slang@67dfde8) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix selection order of prefix/postfix AST fields - [#857](NomicFoundation/slang#857) [`f677d5e`](NomicFoundation/slang@f677d5e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `FieldName` to `NodeLabel` - [#852](NomicFoundation/slang#852) [`ca79eca`](NomicFoundation/slang@ca79eca) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow parsing `ColonEqual` as two separate tokens before `0.5.5` - [#889](NomicFoundation/slang#889) [`ce5050f`](NomicFoundation/slang@ce5050f) Thanks [@Xanewok](https://github.com/Xanewok)! - Support `delete` as an expression rather than a statement - [#923](NomicFoundation/slang#923) [`bb30fc1`](NomicFoundation/slang@bb30fc1) Thanks [@Xanewok](https://github.com/Xanewok)! - Support arbitrary ASCII escape sequences in string literals until 0.4.25 - [#887](NomicFoundation/slang#887) [`dff1201`](NomicFoundation/slang@dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Support view and pure function modifiers only from 0.4.16 - [#800](NomicFoundation/slang#800) [`d1827ff`](NomicFoundation/slang@d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `AsciiStringLiteral` to `StringLiteral` - [#838](NomicFoundation/slang#838) [`ad98d1c`](NomicFoundation/slang@ad98d1c) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - upgrade to rust `1.76.0` - [#849](NomicFoundation/slang#849) [`5c42e0e`](NomicFoundation/slang@5c42e0e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `override` and `virtual` to `ConstructorAttribute` - [#862](NomicFoundation/slang#862) [`5e37ea0`](NomicFoundation/slang@5e37ea0) Thanks [@Xanewok](https://github.com/Xanewok)! - allow call options as a post-fix expression - [#786](NomicFoundation/slang#786) [`0bfa6b7`](NomicFoundation/slang@0bfa6b7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support Yul label statements before `0.5.0` - [#839](NomicFoundation/slang#839) [`2d698eb`](NomicFoundation/slang@2d698eb) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support string literals in version pragmas - [#891](NomicFoundation/slang#891) [`70c9d7d`](NomicFoundation/slang@70c9d7d) Thanks [@Xanewok](https://github.com/Xanewok)! - Fix parsing `<NUMBER>.member` member access expression - [#842](NomicFoundation/slang#842) [`2069126`](NomicFoundation/slang@2069126) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `private` to `UnnamedFunctionAttribute` till `0.5.0` - [#840](NomicFoundation/slang#840) [`7fb0d20`](NomicFoundation/slang@7fb0d20) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow `var` in `TupleDeconstructionStatement` before `0.5.0` Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Xanewok changed the title ~~Trivia as tokens~~ Emit trivia as sibling tokens and don't generate a RuleKind for it Mar 5, 2024

Xanewok marked this pull request as ready for review March 5, 2024 20:36

Xanewok requested a review from a team as a code owner March 5, 2024 20:36

Xanewok force-pushed the trivia-as-tokens branch from 50da88c to ffb40a5 Compare March 6, 2024 10:26

Xanewok mentioned this pull request Mar 7, 2024

Add TokenKind::is_trivia and use it where appropriate #886

Merged

OmarTawfik approved these changes Mar 7, 2024

View reviewed changes

OmarTawfik reviewed Mar 7, 2024

View reviewed changes

Xanewok added 4 commits March 7, 2024 23:05

refactor!: Flatten the trivia into sibling terminal nodes

a884475

refactor!: Stop generating RuleKinds for Leading/TrailingTrivia

6d5ff1b

refactor: Generate the trivia parser functions separately

59f5532

refactor!: Label the trivia terminal types accordingly

52db9c6

Xanewok force-pushed the trivia-as-tokens branch from ffb40a5 to 52db9c6 Compare March 7, 2024 22:06

Xanewok added this pull request to the merge queue Mar 7, 2024

Add a changeset file

341f04d

Xanewok removed this pull request from the merge queue due to a manual request Mar 7, 2024

Xanewok enabled auto-merge March 7, 2024 22:18

Xanewok added this pull request to the merge queue Mar 7, 2024

Merged via the queue into NomicFoundation:main with commit a9bd8da Mar 7, 2024

Xanewok deleted the trivia-as-tokens branch March 7, 2024 22:32

github-actions bot mentioned this pull request Mar 7, 2024

Bump Slang Version #783

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Emit trivia as sibling tokens and don't generate a `RuleKind` for it#885

Emit trivia as sibling tokens and don't generate a `RuleKind` for it#885
Xanewok merged 5 commits intoNomicFoundation:mainfrom
Xanewok:trivia-as-tokens

Xanewok commented Mar 5, 2024

Uh oh!

changeset-bot bot commented Mar 5, 2024 •

edited

Loading

Uh oh!

Xanewok commented Mar 5, 2024

Uh oh!

OmarTawfik Mar 7, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -45,12 +45,12 @@ impl GrammarConstructorDslV2 for Grammar {

		let leading_trivia = Rc::new(NamedTriviaParser {
		name: "LeadingTrivia",

Conversation

Xanewok commented Mar 5, 2024

Uh oh!

changeset-bot bot commented Mar 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Xanewok commented Mar 5, 2024

Uh oh!

OmarTawfik Mar 7, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

changeset-bot bot commented Mar 5, 2024 •

edited

Loading