<regex>: Properly parse backslashes in character classes of basic regexes#5523
Merged
StephanTLavavej merged 21 commits intomicrosoft:mainfrom May 22, 2025
Merged
Conversation
…arse flag for forbidden dash at range start
Member
|
Reviewing now, I'll push changes soon. I updated the PR description from saying " |
Use muellerj2's superior descriptions of `_L_alt_nl` and `_L_no_nl`. Note that `_L_no_nl` is (grep, egrep). Note that `_L_esc_oct` and `_L_esc_ffnx` are (awk). `_L_esc_ffn` confusingly said "(\[fnrtv])" when other comments like `_L_ident_ERE` mean square brackets literally. Spell out "(\f \n \r \t \v)" for clarity and improved searchability. Rephrase `_L_ident_awk`'s comment for clarity. Note that `_L_anch_rstr` is (BRE) only, `_L_paren_bal` is (ERE) only, and `_L_brk_rstr` is (ERE, BRE).
StephanTLavavej
approved these changes
May 22, 2025
Member
|
Thanks!! 😻 I pushed some follow-up commits for additional cleanups, please double-check. I really appreciate the well-structured commit history here; ordinarily I would be nervous about mixing a refactoring and a bugfix but this was entirely reasonable. |
Member
|
I'm mirroring this to the MSVC-internal repo - please notify me if any further changes are pushed. |
Member
|
Thanks for the infinite bugfixes in infinite combinations, as the Vulcans say! 🖖 🐞 🛠️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5379.
This renames the parser to add a new member variable describing the lexer mode: Default or inside character class. This allows the lexer to correctly process a backslash when parsing a character class/bracket expression.
I also tried to do this without renaming the parser, but this would mean we would have to pass the lexer mode (in or outside a character class) as an argument to all the functions processing escapes in any way, which is a bit of a pain. By renaming the parser, we need the least changes to the logic itself.
Since the parser is renamed, this PR is also doing a number of minor cleanups to the parser and builder (which is also renamed to do these cleanups).
The PR is split into several commits to simplify reviewing:
_Parserto_Parser2._Compile. Specifically:_L_brk_balis assigned its own bit; previously it wasSTL/stl/inc/regex
Line 1879 in cbd091e
_L_grp_escflag is added to the awk flags so that the workaround in_ClassAtomcan be removed._Lang_flagsenum and the_L_flagsmember variable tounsigned long longso that we can add more flags more easily in the future. (This already adds_L_dsh_rstrto signify that the dash-cannot appear as the starting point of a character range in BREs and EREs, but doesn't perform the parser changes to support it yet.)_Beginfrom the parser._Charis usually acharorwchar_t, so it [plus the single-byte_Modemember variable added in the last commit] can usually fit into the four bytes the compiler must add after_Mchar._Builderto_Builder2._Bmaxand_Tmaxfrom the builder.<regex>: Backslashes in character classes are sometimes not matched in basic regular expressions #5379 essentially by making_Is_esc()always return false when not in default (read: outside-bracketed-character-class) mode. Note that it matters how we change the lexer mode in_Parser2::_Alternative():_Next()and_Expect()process the first token inside or outside the square brackets, so we must change the mode before calling these functions. The tests check that we didn't get this wrong.