Utf16.IsValid and Utf8/16.IndexOfInvalidSubsequence by lilinus · Pull Request #120326 · dotnet/runtime

lilinus · 2025-10-02T14:24:49Z

Copilot

Pull Request Overview

This PR adds new UTF-16 validation APIs to System.Text.Unicode, introducing Utf16.IsValid and Utf16.IndexOfInvalidSubsequence methods alongside similar functionality for UTF-8. It also refactors existing code to use these new standardized validation APIs instead of custom validation logic.

Adds new Utf16 class with validation methods IsValid and IndexOfInvalidSubsequence
Adds Utf8.IndexOfInvalidSubsequence method to complement existing Utf8.IsValid
Refactors multiple codebases to use the new standardized validation APIs

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
System.Private.CoreLib.Shared.projitems	Adds the new Utf16.cs file to the build
Utf16.cs	New class implementing UTF-16 validation methods
Utf8.cs	Adds IndexOfInvalidSubsequence method and updates class documentation
System.Runtime.cs	Adds public API surface for new UTF-16 and UTF-8 validation methods
HttpEncoder.cs	Refactors surrogate validation to use new Utf16.IndexOfInvalidSubsequence
Normalization.Icu.cs	Replaces custom validation logic with Utf16.IsValid
StringSearchValues.cs	Simplifies surrogate validation using Utf16.IsValid
Utf8UtilityTests.ValidateBytes.cs	Adds test coverage for new Utf8.IndexOfInvalidSubsequence method
Utf16UtilityTests.ValidateChars.cs	Adds test coverage for new Utf16 validation methods

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

dotnet-policy-service · 2025-10-27T08:17:40Z

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

jeffhandley · 2026-02-01T19:00:55Z

@lilinus Thanks for your patience on this; we are planning to review it this month. I've updated the branch and CI will re-run, but there was an ApiCompat error in the build previously indicating Utf8.IndexOfInvalidSubsequence wasn't available in the net8.0 target and it wasn't obvious to me why that was the case.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

Update XML doc comments and remove unneeded using statements. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

...tem.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

jozkee

Other than the unsafe comment, this LGTM.

Is worth mentioning that

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/Normalization.Icu.cs

Line 221 in 167d862

private static bool HasInvalidUnicodeSequence(ReadOnlySpan<char> s)

from #118018 can't adopt the new APIs because it would make it more expensive by requiring two passes instead of one.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16Utility.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

lilinus · 2026-02-28T07:32:36Z

I'm getting some compat errors in Microsoft.Bcl.Memory.

Not sure how to approach those unfortunately. Should the new methods be excluded or included in that package?

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs:830

The XML docs for Utf8.IsValid immediately below this new API still describe the input as a "ReadOnlySpan string". Now that IndexOfInvalidSubsequence is added, please update Utf8.IsValid's <param name="value"> documentation to describe validating a UTF-8 byte sequence / input text, for consistency and correctness.

            Utf8Utility.GetIndexOfFirstInvalidUtf8Sequence(value, out _);
#endif

        /// <summary>
        /// Validates that the value is well-formed UTF-8.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs:834

The XML docs for Utf8.IsValid still describe the input parameter as a generic "ReadOnlySpan string", but this API validates a UTF-8 byte sequence (ReadOnlySpan). Please update the text so public docs accurately describe the expected input, ideally matching the wording used for IndexOfInvalidSubsequence just above.

        /// <summary>
        /// Validates that the value is well-formed UTF-8.
        /// </summary>
        /// <param name="value">The <see cref="ReadOnlySpan{T}"/> string.</param>
        /// <returns><c>true</c> if value is well-formed UTF-8, <c>false</c> otherwise.</returns>
        public static bool IsValid(ReadOnlySpan<byte> value) =>

lilinus added 5 commits September 29, 2025 14:57

Add methods

3b6fdfb

Add tests

878724a

Use Utf16 CoreLib

9705165

Use helper in HttpEncoder

e2439c5

Add removed line ending

5edcb5c

Copilot AI review requested due to automatic review settings October 2, 2025 14:24

github-actions bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 2, 2025

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Oct 2, 2025

Copilot AI reviewed Oct 2, 2025

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs Outdated Show resolved Hide resolved

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Oct 2, 2025

[android] Android.Device_Emulator.JIT.Test failing on emulators with CoreCLR #112633

Open

lilinus added 2 commits October 3, 2025 09:28

Revert Normalization.Icu.cs change

307404b

Address copilot feedback

e0868b9

This was referenced Oct 3, 2025

STATUS_UNSUCCESSFUL in RsaCryptRoundtrip_OaepSHA1 #29683

Open

RSA 384 failing in Windows #120353

Closed

MihuBot mentioned this pull request Oct 7, 2025

[✨ Triage] dotnet/runtime#120499 by ericstj - System.Net docs for .NET 10.0 APIs MihuBot/runtime-utils#1562

Closed

IDisposable approved these changes Oct 18, 2025

View reviewed changes

marek-safar added the area-System.Runtime label Oct 27, 2025

jeffhandley removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jan 14, 2026

jeffhandley assigned jozkee Jan 14, 2026

Merge branch 'main' into utf8-utf16-validation

a0bd5e9

Copilot AI review requested due to automatic review settings February 1, 2026 18:58

Copilot started reviewing on behalf of jeffhandley February 1, 2026 18:59 View session

Copilot AI reviewed Feb 1, 2026

View reviewed changes

Apply suggestions from code review

b6cd454

Update XML doc comments and remove unneeded using statements. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 1, 2026 19:32

Copilot started reviewing on behalf of jeffhandley February 1, 2026 19:32 View session

Copilot AI reviewed Feb 1, 2026

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs Outdated Show resolved Hide resolved

...tem.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Feb 1, 2026

Cannot find 'arm64-v8a' device dotnet/dnceng#2284

Open

3 tasks

Copilot AI reviewed Feb 16, 2026

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs Show resolved Hide resolved

Rename ContainsIncompleteSurrogatePairs

e6e4b4e

build-analysis bot mentioned this pull request Feb 16, 2026

crossgen2 failing on Linux Arm32 intermittently #124007

Closed

jozkee reviewed Feb 25, 2026

View reviewed changes

Internal unsafe method

6839cba

Copilot AI review requested due to automatic review settings February 27, 2026 09:32

Copilot started reviewing on behalf of lilinus February 27, 2026 09:33 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16Utility.cs Outdated Show resolved Hide resolved

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs Outdated Show resolved Hide resolved

Apply copilot suggestion

90be926

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 27, 2026 09:45

Copilot started reviewing on behalf of lilinus February 27, 2026 09:46 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

Only compile internal method for SPC

95b2b20

Exclude methods in OOB package

e5b2527

Copilot AI review requested due to automatic review settings March 2, 2026 13:10

Merge branch 'main' into utf8-utf16-validation

afa99e9

Copilot started reviewing on behalf of lilinus March 2, 2026 13:13 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf16.cs Outdated Show resolved Hide resolved

Apply suggestion from @Copilot

af26d02

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 2, 2026 14:00

Copilot started reviewing on behalf of lilinus March 2, 2026 14:01 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

This was referenced Mar 2, 2026

BinderTracingTest.ResolutionFlow times out #104670

Open

Test failure: baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd #110173

Open

Android WebSocket failure #121518

Open

Conversation

lilinus commented Oct 2, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

dotnet-policy-service bot commented Oct 27, 2025

Uh oh!

jeffhandley commented Feb 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

jozkee left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

lilinus commented Feb 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants