Skip to content

UnmanagedCallersOnly does not work when called from a P/Invoke in Release mode #38192

@tannergooding

Description

@tannergooding

As per the title, the following program succeeds if run under Debug but fails under Release

using System;
using System.Runtime.InteropServices;

unsafe class Program
{
    public const int CS_VREDRAW = 0x0001;
    public const int CS_HREDRAW = 0x0002;

    public const int CW_USEDEFAULT = unchecked((int)0x80000000);

    public const int SW_SHOWDEFAULT = 10;

    public const uint WS_OVERLAPPED = 0x00000000;
    public const uint WS_CAPTION = 0x00C00000;
    public const uint WS_SYSMENU = 0x00080000;
    public const uint WS_THICKFRAME = 0x00040000;
    public const uint WS_MINIMIZEBOX = 0x00020000;
    public const uint WS_MAXIMIZEBOX = 0x00010000;

    public const uint WS_OVERLAPPEDWINDOW = WS_OVERLAPPED | WS_CAPTION | WS_SYSMENU | WS_THICKFRAME | WS_MINIMIZEBOX | WS_MAXIMIZEBOX;

    public unsafe partial struct WNDCLASSEXW
    {
        public uint cbSize;
        public uint style;
        public delegate* stdcall<IntPtr, uint, nuint, nint, nint> lpfnWndProc;
        public int cbClsExtra;
        public int cbWndExtra;
        public IntPtr hInstance;
        public IntPtr hIcon;
        public IntPtr hCursor;
        public IntPtr hbrBackground;
        public ushort* lpszMenuName;
        public ushort* lpszClassName;
        public IntPtr hIconSm;
    }

    [DllImport("user32", EntryPoint = "CreateWindowExW", ExactSpelling = true)]
    public static extern IntPtr CreateWindowExW(uint dwExStyle, ushort* lpClassName, ushort* lpWindowName, uint dwStyle, int X, int Y, int nWidth, int nHeight, IntPtr hWndParent, IntPtr hMenu, IntPtr hInstance, void* lpParam);

    [DllImport("user32", EntryPoint = "DefWindowProcW", ExactSpelling = true)]
    public static extern nint DefWindowProcW(IntPtr hWnd, uint Msg, nuint wParam, nint lParam);

    [DllImport("user32", EntryPoint = "RegisterClassExW", ExactSpelling = true)]
    public static extern ushort RegisterClassExW(WNDCLASSEXW* param0);

    [UnmanagedCallersOnly]
    private static nint WindowProc(IntPtr hWnd, uint Msg, nuint wParam, nint lParam)
    {
        return DefWindowProcW(hWnd, Msg, wParam, lParam);
    }

    static void Main(string[] args)
    {
        fixed (char* lpszClassName = "SampleClass")
        fixed (char* lpWindowName = "SampleTitle")
        {
            var hInstance = Marshal.GetHINSTANCE(typeof(Program).Module);

            // Requires an explicit cast until C# handles UnmanagedCallersOnly
            var wndProc = (delegate* stdcall<IntPtr, uint, nuint, nint, nint>)(delegate* managed<IntPtr, uint, nuint, nint, nint>)&WindowProc;

            // Initialize the window class.
            var windowClass = new WNDCLASSEXW
            {
                cbSize = (uint)sizeof(WNDCLASSEXW),
                style = CS_HREDRAW | CS_VREDRAW,
                lpfnWndProc = wndProc,
                hInstance = hInstance,
                lpszClassName = (ushort*)lpszClassName
            };
            _ = RegisterClassExW(&windowClass);

            var hwnd = CreateWindowExW(
                0,
                windowClass.lpszClassName,
                (ushort*)lpWindowName,
                WS_OVERLAPPEDWINDOW,
                X: CW_USEDEFAULT,
                Y: CW_USEDEFAULT,
                nWidth: CW_USEDEFAULT,
                nHeight: CW_USEDEFAULT,
                hWndParent: IntPtr.Zero,
                hMenu: IntPtr.Zero,
                hInstance,
                lpParam: null
            );
        }
    }
}

The csproj should resemble the following so function pointers can be used:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
    <LangVersion>preview</LangVersion>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
  </PropertyGroup>

  <PropertyGroup>
    <RestoreSources>
      https://api.nuget.org/v3/index.json;
      https://dotnet.myget.org/F/roslyn/api/v3/index.json;
    </RestoreSources>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Net.Compilers.Toolset" Version="3.8.0-1.20320.1" />
  </ItemGroup>

</Project>

A more complete program will successfully display the window, handle the callbacks etc in Debug. Additionally it will all work as expected under Release if using Marshal.GetFunctionPointerForDelegate instead.

However, under Release when using UnmanagedCallersOnly, it fails with the following:

Fatal error. Invalid Program: attempted to call a UnmanagedCallersOnly method from managed code.
Repeat 2 times:
--------------------------------
   at Program.CreateWindowExW(UInt32, UInt16*, UInt16*, UInt32, Int32, Int32, Int32, Int32, IntPtr, IntPtr, IntPtr, Void*)
--------------------------------
   at Program.Main(System.String[])

Metadata

Metadata

Assignees

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIbug

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions