parse line breaks without newlines by OmarTawfik · Pull Request #854 · NomicFoundation/slang

OmarTawfik · 2024-02-24T12:11:04Z

No description provided.

changeset-bot · 2024-02-24T12:11:07Z

🦋 Changeset detected

Latest commit: ee9122f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@nomicfoundation/slang	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Xanewok · 2024-02-28T14:15:43Z

Bare CR are recognized as newlines at least for the multi-line comments starting from argotorg/solidity@f85f6ba#diff-16d471fed3639e7154b8661355cf66d41efe54a05f2d8e67017b9a301d19f7cd, so I'm happy to merge this.

It seems that solc also respects these for line breaking the strings/single line (doc) comments:

bool Scanner::isUnicodeLinebreak()
{
	if (0x0a <= m_char && m_char <= 0x0d)
		// line feed, vertical tab, form feed, carriage return
		return true;
	if (!m_source.isPastEndOfInput(1) && uint8_t(m_source.get(0)) == 0xc2 && uint8_t(m_source.get(1)) == 0x85)
		// NEL - U+0085, C2 85 in utf8
		return true;
	if (!m_source.isPastEndOfInput(2) && uint8_t(m_source.get(0)) == 0xe2 && uint8_t(m_source.get(1)) == 0x80 && (
		uint8_t(m_source.get(2)) == 0xa8 || uint8_t(m_source.get(2)) == 0xa9
	))
		// LS - U+2028, E2 80 A8  in utf8
		// PS - U+2029, E2 80 A9  in utf8
		return true;
	return false;
}

@AntonyBlakey

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @nomicfoundation/slang@0.14.0 ### Minor Changes - [#753](#753) [`b35c763`](b35c763) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Add tree query implementation as `Query::parse` and `Cursor::query` - [#755](#755) [`8c260fc`](8c260fc) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support parsing NatSpec comments - [#908](#908) [`ab3688b`](ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Changed the cst.NodeType in TS to use more descriptive string values rather than 0/1 integers - [#886](#886) [`0125717`](0125717) Thanks [@Xanewok](https://github.com/Xanewok)! - Add `TokenKind::is_trivia` - [#887](#887) [`dff1201`](dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Add support for constant function modifier removed in 0.5.0 - [#885](#885) [`a9bd8da`](a9bd8da) Thanks [@Xanewok](https://github.com/Xanewok)! - Flatten the trivia syntax nodes into sibling tokens - [#908](#908) [`ab3688b`](ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Add RuleNode/TokenNode::toJSON() in the TypeScript API ### Patch Changes - [#801](#801) [`ecbba49`](ecbba49) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - unreserve pragma keywords in all versions - [#869](#869) [`951b58d`](951b58d) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support dots in yul identifiers from `0.5.8` till `0.7.0` - [#890](#890) [`1ff8599`](1ff8599) Thanks [@Xanewok](https://github.com/Xanewok)! - Mark `override` as being a valid attribute only after 0.6.0 - [#800](#800) [`d1827ff`](d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support unicode characters in string literals up to `0.7.0` - [#797](#797) [`86f36d7`](86f36d7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix source locations for unicode characters in error reports - [#854](#854) [`4b8970b`](4b8970b) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - parse line breaks without newlines - [#844](#844) [`f62de9e`](f62de9e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing empty `/**/` comments - [#799](#799) [`303dda9`](303dda9) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prevent parsing multiple literals under `StringExpression` before `0.5.14` - [#847](#847) [`6b6f260`](6b6f260) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prioritize parsing `MultiLineComment` over `MultiLineNatSpecComment` - [#796](#796) [`59e1e53`](59e1e53) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `public` and `internal` to `UnnamedFunctionAttribute` till `0.5.0` - [#756](#756) [`e839817`](e839817) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing `payable` primary expressions - [#851](#851) [`67dfde8`](67dfde8) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix selection order of prefix/postfix AST fields - [#857](#857) [`f677d5e`](f677d5e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `FieldName` to `NodeLabel` - [#852](#852) [`ca79eca`](ca79eca) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow parsing `ColonEqual` as two separate tokens before `0.5.5` - [#889](#889) [`ce5050f`](ce5050f) Thanks [@Xanewok](https://github.com/Xanewok)! - Support `delete` as an expression rather than a statement - [#923](#923) [`bb30fc1`](bb30fc1) Thanks [@Xanewok](https://github.com/Xanewok)! - Support arbitrary ASCII escape sequences in string literals until 0.4.25 - [#887](#887) [`dff1201`](dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Support view and pure function modifiers only from 0.4.16 - [#800](#800) [`d1827ff`](d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `AsciiStringLiteral` to `StringLiteral` - [#838](#838) [`ad98d1c`](ad98d1c) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - upgrade to rust `1.76.0` - [#849](#849) [`5c42e0e`](5c42e0e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `override` and `virtual` to `ConstructorAttribute` - [#862](#862) [`5e37ea0`](5e37ea0) Thanks [@Xanewok](https://github.com/Xanewok)! - allow call options as a post-fix expression - [#786](#786) [`0bfa6b7`](0bfa6b7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support Yul label statements before `0.5.0` - [#839](#839) [`2d698eb`](2d698eb) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support string literals in version pragmas - [#891](#891) [`70c9d7d`](70c9d7d) Thanks [@Xanewok](https://github.com/Xanewok)! - Fix parsing `<NUMBER>.member` member access expression - [#842](#842) [`2069126`](2069126) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `private` to `UnnamedFunctionAttribute` till `0.5.0` - [#840](#840) [`7fb0d20`](7fb0d20) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow `var` in `TupleDeconstructionStatement` before `0.5.0` Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

@AntonyBlakey

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @nomicfoundation/slang@0.14.0 ### Minor Changes - [#753](NomicFoundation/slang#753) [`b35c763`](NomicFoundation/slang@b35c763) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Add tree query implementation as `Query::parse` and `Cursor::query` - [#755](NomicFoundation/slang#755) [`8c260fc`](NomicFoundation/slang@8c260fc) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support parsing NatSpec comments - [#908](NomicFoundation/slang#908) [`ab3688b`](NomicFoundation/slang@ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Changed the cst.NodeType in TS to use more descriptive string values rather than 0/1 integers - [#886](NomicFoundation/slang#886) [`0125717`](NomicFoundation/slang@0125717) Thanks [@Xanewok](https://github.com/Xanewok)! - Add `TokenKind::is_trivia` - [#887](NomicFoundation/slang#887) [`dff1201`](NomicFoundation/slang@dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Add support for constant function modifier removed in 0.5.0 - [#885](NomicFoundation/slang#885) [`a9bd8da`](NomicFoundation/slang@a9bd8da) Thanks [@Xanewok](https://github.com/Xanewok)! - Flatten the trivia syntax nodes into sibling tokens - [#908](NomicFoundation/slang#908) [`ab3688b`](NomicFoundation/slang@ab3688b) Thanks [@Xanewok](https://github.com/Xanewok)! - Add RuleNode/TokenNode::toJSON() in the TypeScript API ### Patch Changes - [#801](NomicFoundation/slang#801) [`ecbba49`](NomicFoundation/slang@ecbba49) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - unreserve pragma keywords in all versions - [#869](NomicFoundation/slang#869) [`951b58d`](NomicFoundation/slang@951b58d) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support dots in yul identifiers from `0.5.8` till `0.7.0` - [#890](NomicFoundation/slang#890) [`1ff8599`](NomicFoundation/slang@1ff8599) Thanks [@Xanewok](https://github.com/Xanewok)! - Mark `override` as being a valid attribute only after 0.6.0 - [#800](NomicFoundation/slang#800) [`d1827ff`](NomicFoundation/slang@d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support unicode characters in string literals up to `0.7.0` - [#797](NomicFoundation/slang#797) [`86f36d7`](NomicFoundation/slang@86f36d7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix source locations for unicode characters in error reports - [#854](NomicFoundation/slang#854) [`4b8970b`](NomicFoundation/slang@4b8970b) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - parse line breaks without newlines - [#844](NomicFoundation/slang#844) [`f62de9e`](NomicFoundation/slang@f62de9e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing empty `/**/` comments - [#799](NomicFoundation/slang#799) [`303dda9`](NomicFoundation/slang@303dda9) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prevent parsing multiple literals under `StringExpression` before `0.5.14` - [#847](NomicFoundation/slang#847) [`6b6f260`](NomicFoundation/slang@6b6f260) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - prioritize parsing `MultiLineComment` over `MultiLineNatSpecComment` - [#796](NomicFoundation/slang#796) [`59e1e53`](NomicFoundation/slang@59e1e53) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `public` and `internal` to `UnnamedFunctionAttribute` till `0.5.0` - [#756](NomicFoundation/slang#756) [`e839817`](NomicFoundation/slang@e839817) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix parsing `payable` primary expressions - [#851](NomicFoundation/slang#851) [`67dfde8`](NomicFoundation/slang@67dfde8) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fix selection order of prefix/postfix AST fields - [#857](NomicFoundation/slang#857) [`f677d5e`](NomicFoundation/slang@f677d5e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `FieldName` to `NodeLabel` - [#852](NomicFoundation/slang#852) [`ca79eca`](NomicFoundation/slang@ca79eca) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow parsing `ColonEqual` as two separate tokens before `0.5.5` - [#889](NomicFoundation/slang#889) [`ce5050f`](NomicFoundation/slang@ce5050f) Thanks [@Xanewok](https://github.com/Xanewok)! - Support `delete` as an expression rather than a statement - [#923](NomicFoundation/slang#923) [`bb30fc1`](NomicFoundation/slang@bb30fc1) Thanks [@Xanewok](https://github.com/Xanewok)! - Support arbitrary ASCII escape sequences in string literals until 0.4.25 - [#887](NomicFoundation/slang#887) [`dff1201`](NomicFoundation/slang@dff1201) Thanks [@Xanewok](https://github.com/Xanewok)! - Support view and pure function modifiers only from 0.4.16 - [#800](NomicFoundation/slang#800) [`d1827ff`](NomicFoundation/slang@d1827ff) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - rename `AsciiStringLiteral` to `StringLiteral` - [#838](NomicFoundation/slang#838) [`ad98d1c`](NomicFoundation/slang@ad98d1c) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - upgrade to rust `1.76.0` - [#849](NomicFoundation/slang#849) [`5c42e0e`](NomicFoundation/slang@5c42e0e) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `override` and `virtual` to `ConstructorAttribute` - [#862](NomicFoundation/slang#862) [`5e37ea0`](NomicFoundation/slang@5e37ea0) Thanks [@Xanewok](https://github.com/Xanewok)! - allow call options as a post-fix expression - [#786](NomicFoundation/slang#786) [`0bfa6b7`](NomicFoundation/slang@0bfa6b7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support Yul label statements before `0.5.0` - [#839](NomicFoundation/slang#839) [`2d698eb`](NomicFoundation/slang@2d698eb) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - support string literals in version pragmas - [#891](NomicFoundation/slang#891) [`70c9d7d`](NomicFoundation/slang@70c9d7d) Thanks [@Xanewok](https://github.com/Xanewok)! - Fix parsing `<NUMBER>.member` member access expression - [#842](NomicFoundation/slang#842) [`2069126`](NomicFoundation/slang@2069126) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - add `private` to `UnnamedFunctionAttribute` till `0.5.0` - [#840](NomicFoundation/slang#840) [`7fb0d20`](NomicFoundation/slang@7fb0d20) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - allow `var` in `TupleDeconstructionStatement` before `0.5.0` Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

parse line breaks without newlines

ee9122f

OmarTawfik force-pushed the line-breaks branch from a0cbf9c to ee9122f Compare February 24, 2024 13:08

OmarTawfik marked this pull request as ready for review February 26, 2024 16:58

OmarTawfik requested a review from a team as a code owner February 26, 2024 16:58

OmarTawfik enabled auto-merge February 26, 2024 16:58

Xanewok approved these changes Feb 28, 2024

View reviewed changes

OmarTawfik added this pull request to the merge queue Feb 28, 2024

Merged via the queue into NomicFoundation:main with commit 4b8970b Feb 28, 2024

OmarTawfik deleted the line-breaks branch February 28, 2024 14:24

github-actions bot mentioned this pull request Feb 28, 2024

Bump Slang Version #783

Merged

Xanewok mentioned this pull request Mar 9, 2024

Fully parse the smart contract sanctuary ethereum/mainnet dataset #892

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parse line breaks without newlines#854

parse line breaks without newlines#854
OmarTawfik merged 1 commit intoNomicFoundation:mainfrom
OmarTawfik-forks:line-breaks

OmarTawfik commented Feb 24, 2024

Uh oh!

changeset-bot bot commented Feb 24, 2024 •

edited

Loading

Uh oh!

Xanewok commented Feb 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

OmarTawfik commented Feb 24, 2024

Uh oh!

changeset-bot bot commented Feb 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Xanewok commented Feb 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

changeset-bot bot commented Feb 24, 2024 •

edited

Loading