Skip to content

Fix RegexParser to ignore vertical tabs with RegexOptions.IgnorePatternWhitespace#120625

Merged
stephentoub merged 3 commits intomainfrom
copilot/fix-regex-vertical-tab-issue
Oct 17, 2025
Merged

Fix RegexParser to ignore vertical tabs with RegexOptions.IgnorePatternWhitespace#120625
stephentoub merged 3 commits intomainfrom
copilot/fix-regex-vertical-tab-issue

Conversation

Copy link
Contributor

Copilot AI commented Oct 11, 2025 •
edited
Loading

Fix: Regex parser to ignore vertical tabs in pattern with RegexOptions.IgnorePatternWhitespace

Plan:

  • Investigate the issue and reproduce the bug
  • Build baseline (clr+libs) successfully
  • Confirm the bug: vertical tab (\v, 0xB) is not ignored with IgnorePatternWhitespace
  • Fix the Category array in RegexParser.cs by changing position 0xB from 0 to W
  • Update s_metachars to include vertical tab character
  • Add test case to existing test file (Regex.Match.Tests.cs)
  • Build and test the fix
  • Verify all tests pass
  • Guard test for .NET Framework platform

Changes Made:

  1. RegexParser.cs (line 1932): Changed Category[0xB] from 0 to W to mark vertical tab as whitespace
  2. RegexParser.cs (line 1942): Added \v to s_metachars SearchValues
  3. Regex.Match.Tests.cs (line 584-588): Added test case for vertical tab with IgnorePatternWhitespace option, guarded with if (!PlatformDetection.IsNetFramework) to prevent failures on .NET Framework

Test Results:

✅ FunctionalTests: 29,291 tests passed, 0 failures
✅ UnitTests: 1,005 tests passed, 0 failures
✅ New test case: Verified passing across all regex engines (Interpreter, Compiled, NonBacktracking, SourceGenerated) on .NET Core

Technical Details:

  • Position 0xB in the Category array corresponds to the vertical tab character ('\v')
  • The fix aligns with .NET's char.IsWhiteSpace('\v') which returns true
  • PCRE2 and other regex engines also treat vertical tab as whitespace in ignore pattern whitespace mode
  • The vertical tab character is now properly ignored when using RegexOptions.IgnorePatternWhitespace on .NET Core
  • Test is guarded to not run on .NET Framework where the fix is not applicable
Original prompt

This section details on the original issue you should resolve

<issue_title>Regex parser doesn't ignore vertical tabs in patter when using RegexOptions.IgnorePatternWhiteSpace</issue_title>
<issue_description>While porting PCRE2 tests suite, one of the failing tests is:

bool isMatch = Regex.IsMatch("ab", "a\vb", RegexOptions.IgnorePatternWhitespace);
Assert.True(isMatch);

This should probably be true and match other engines like PCRE as \v character should be ignored as whitespace and just have the pattern be ab

cc: @stephentoub </issue_description>

Comments on the Issue (you are @copilot in this section)

@ Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions See info in [area-owners.md](https://github.com/dotnet/runtime/blob/main/docs/area-owners.md) if you want to be subscribed.
Issue Details

While porting PCRE2 tests suite, one of the failing tests is:

bool isMatch = Regex.IsMatch("ab", "a\vb", RegexOptions.IgnorePatternWhitespace);
Assert.True(isMatch);

This should probably be true and match other engines like PCRE as \v character should be ignored as whitespace and just have the pattern be ab

cc: @stephentoub

Author: joperezr
Assignees: -
Labels:

area-System.Text.RegularExpressions
Milestone: -
@stephentoub This does seem like a bug; `char.IsWhitespace('\v')` is true, as is `Regex.IsMatch("\v", @"\s")`. Presumably it'd be as simple as changing the `Category[0xB]` value in the following table to be 'X': https://github.com/dotnet/runtime/blob/aac729ff906a31f327823587748687c0308a4043/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs#L2060-L2062 We might want to dig a tad deeper, as it feels a little deliberate that cell was left as 0 rather than X.

Fixes #73206

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@stephentoub
Fix vertical tab handling in RegexOptions.IgnorePatternWhitespace
a233d04 
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix regex parser to ignore vertical tabs with IgnorePatternWhiteSpace Fix RegexParser to ignore vertical tabs with RegexOptions.IgnorePatternWhitespace Oct 11, 2025
Copilot AI requested a review from stephentoub October 11, 2025 16:55
@stephentoub stephentoub marked this pull request as ready for review October 11, 2025 18:00
Copilot AI review requested due to automatic review settings October 11, 2025 18:00
Copilot AI reviewed Oct 11, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes a bug in the regex parser where vertical tab (\v, character 0xB) was not being treated as whitespace when using RegexOptions.IgnorePatternWhitespace, making it inconsistent with other .NET APIs and regex engines like PCRE2.

  • Updated the ASCII character category table to mark vertical tab as whitespace
  • Added vertical tab to the metacharacters SearchValues for proper escaping
  • Added test case to verify vertical tab is properly ignored with IgnorePatternWhitespace option

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
RegexParser.cs Updated character category table to treat vertical tab as whitespace and added \v to metacharacters
Regex.Match.Tests.cs Added test case verifying vertical tab is ignored with IgnorePatternWhitespace option

@stephentoub
Guard vertical tab test for netfx platform
c5ec742 
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
Copilot AI requested a review from stephentoub October 11, 2025 18:39
@stephentoub stephentoub enabled auto-merge (squash) October 11, 2025 18:42
@stephentoub stephentoub requested a review from joperezr October 11, 2025 18:42
@jeffhandley jeffhandley requested a review from tarekgh October 17, 2025 17:23
tarekgh
tarekgh approved these changes Oct 17, 2025
View reviewed changes
Copy link
Member

@tarekgh tarekgh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@stephentoub stephentoub merged commit 1ef4c1a into main Oct 17, 2025
88 checks passed
@jkotas jkotas deleted the copilot/fix-regex-vertical-tab-issue branch October 18, 2025 04:00
@dotnet-maestro dotnet-maestro bot mentioned this pull request Oct 21, 2025
Merged
@github-actions github-actions bot locked and limited conversation to collaborators Nov 17, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code review Copilot Copilot left review comments

@joperezr joperezr Awaiting requested review from joperezr

@tarekgh tarekgh Awaiting requested review from tarekgh

+1 more reviewer

@stephentoub stephentoub stephentoub approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoub stephentoub

Copilot code review Copilot

Labels

area-System.Text.RegularExpressions

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Regex parser doesn't ignore vertical tabs in patter when using RegexOptions.IgnorePatternWhiteSpace

4 participants

@stephentoub @tarekgh