Use re-lexing for normal list parsing#11871
Merged
dhruvmanila merged 1 commit intomainfrom Jun 18, 2024
Merged
Conversation
CodSpeed Performance ReportMerging #11871 will not alter performanceComparing Summary
|
Contributor
|
14fcbb3 to
1989946
Compare
65a60fc to
c80a501
Compare
c80a501 to
7543306
Compare
MichaReiser
approved these changes
Jun 18, 2024
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR is a follow-up on #11845 to add the re-lexing logic for normal list parsing.
A normal list parsing is basically parsing elements without any separator in between i.e., there can only be trivia tokens in between the two elements. Currently, this is only being used for parsing assignment statement and f-string elements. Assignment statements cannot be in a parenthesized context, but f-string can have curly braces so this PR is specifically for them.
I don't think this is an ideal recovery but the problem is that both lexer and parser could add an error for f-strings. If the lexer adds an error it'll emit an
Unknowntoken instead while the parser adds the error directly. I think we'd need to move all f-string errors to be emitted by the parser instead. This way the parser can correctly inform the lexer that it's out of an f-string and then the lexer can pop the current f-string context out of the stack.Test Plan
Add test cases, update the snapshots, and run the fuzzer.