Description

In .NET 11 preview.3, LibraryImportDiagnosticsAnalyzer fails to report SYSLIB1051 for StringBuilder parameters when StringMarshalling = StringMarshalling.Utf16 is set on the LibraryImportAttribute. Without StringMarshalling, the error is correctly reported.

This is a regression from preview.2. PR #123780 moved diagnostic reporting from inline in LibraryImportGenerator to a separate LibraryImportDiagnosticsAnalyzer. The generator now discards diagnostics (discardedDiagnostics at LibraryImportGenerator.cs:297), relying on the analyzer to re-derive and report them. The analyzer fails to do so for StringBuilder when StringMarshalling.Utf16 is specified.

SYSLIB1051 is DiagnosticSeverity.Error -- a hard build failure. When the error is not reported, the generator silently produces code that compiles but gives wrong results at runtime (see "Consequence" section below).

Standalone reproduction

Note: you must use the preview.3 SDK specifically. Roslyn analyzer unit tests that run the analyzer from source may not reproduce this, since they do not use the shipped SDK binary.

dotnet-install.ps1 -Version "11.0.100-preview.3.26170.106" -InstallDir "C:\Program Files\dotnet"

global.json -- pin to preview.3:

{
  "sdk": {
    "version": "11.0.100-preview.3.26170.106",
    "allowPrerelease": true
  }
}

Project file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net11.0</TargetFramework>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
  </PropertyGroup>
</Project>

Program.cs:

using System;
using System.Runtime.InteropServices;
using System.Text;

Console.WriteLine("If you see this, SYSLIB1051 was NOT reported.");

static partial class NativeMethods
{
    [LibraryImport("kernel32.dll", StringMarshalling = StringMarshalling.Utf16)]
    internal static partial int GetVolumeNameForVolumeMountPointW(
        string volumeMountPoint,
        [Out] StringBuilder volumeName,
        int bufferLength);
}

Expected: dotnet build fails with SYSLIB1051 ("The type 'System.Text.StringBuilder' is not supported by source-generated P/Invokes...")

Actual with .NET 11 preview.3 SDK (11.0.100-preview.3.26170.106): Build succeeds. No error.

Actual with .NET 11 preview.2 SDK (11.0.100-preview.2.26159.112): SYSLIB1051 correctly reported.

Remove StringMarshalling = StringMarshalling.Utf16 and preview.3 does correctly report SYSLIB1051.

Consequence when the error is missing

When SYSLIB1051 is absent, the generator uses a "forwarder" strategy -- it creates an inner [DllImport] that passes the StringBuilder through to the runtime marshaller. However, the generated inner DllImport omits CharSet = CharSet.Unicode. For W-suffix APIs this means UTF-16 output is marshalled as Ansi: for example, \\?\Volume{guid}\ is read as just \ (0x5C followed by 0x00 is treated as a null-terminated Ansi string).

This is what caused the silent wrong behavior in PR #126660 (commit 8ccebe9), which was caught and fixed in d694082 only because the test happened to fail, not because of any build error.

Why Roslyn analyzer unit tests may not reproduce this

The VerifyAnalyzerAsync tests in LibraryImportGenerator.UnitTests instantiate the analyzer from the locally-built source code, not from the shipped SDK binary. If the shipped preview.3 binary has a bug that the current source on main does not, the unit tests will pass while the standalone repro above fails. The reproduction requires actually using the preview.3 SDK to compile a project.

Docs gap

The official P/Invoke source generation docs do not mention StringBuilder (or CriticalHandle or HandleRef) as unsupported types. The design doc (Compatibility.md) does mention them. It would be useful for the official docs to list the types that trigger SYSLIB1051 so users know what to avoid.

SDK version matrix

Root cause

PR #123780 (commit d97a9c1) introduced LibraryImportDiagnosticsAnalyzer as a separate Roslyn DiagnosticAnalyzer. The generator itself now discards diagnostics. The analyzer's CalculateDiagnostics method recreates SignatureContext and ManagedToNativeStubGenerator to re-derive diagnostics, but the code path for StringBuilder with StringMarshalling.Utf16 fails to produce the expected diagnostic. The exact failure point in the resolver chain was not pinpointed -- the StringMarshallingInfoProvider only handles System.String, not StringBuilder, so StringBuilder should fall through to NotSupportedResolver regardless, but somehow the diagnostic is lost when StringMarshalling is set.