Yeah, I did not test on Mac and the DX unit tests don't work for me, as it always fails to create the DX device. If somebody can give me a hint how to fix that?

I set it up on a Mac to fix the failing unit tests. It's all good now.

On Mac the OpenGL version is currently always 2.1, which means no SM 4 or 5. It's almost certainly related to this.

Yeah, I did not test on Mac and the DX unit tests don't work for me, as it always fails to create the DX device. If somebody can give me a hint how to fix that?

If the device is created with debug flags (DeviceCreationFlags.Debug), you need a feature installed called Graphics Tools when on windows 10 you can find it usually under -> hit win-key -> type "Manage optional features" -> find/add Graphics Tools in there. Lower versions of windows only needed the DXSDK or latest Windows SDK installed.

Thanks for you reply @deccer. I already had the Graphics Tools installed. Looking into it a bit more, the device creation fails because it's trying to create a reference device. If I change it to a regular hardware device, it works.
I'm reading on the internet that this tends to happen on systems with integrated graphics. I have an RX 5700XT, so I shouldn't have this problem. Seems to be a driver problem or something, probably specific to my GPU or system configuration.

For people wanting to try this PR, it's best to use PR #7352 instead. It fully contains this PR, but it has a few additional fixes. Maintaining multiple branches takes some extra time, so I'll only update this branch, if I know it's really needed.

This is awesome! I took a stab at migrating my fairly shader-heavy project, and it works well. Migrating takes a bit of effort though.
I noticed allocating a shader off the mainthread (e.g. calling contentManager.ReadAsset<Effect>(pathToShader, null)) gives GL error 1282 (InvalidOperation). IMHO this might be a good change but previously it was wrapped with a Threading.BlockOnUIThread(() => call.

I also noticed some peculiarities with ShaderConductor:

• ShaderConductor.Compile sometimes attempts to access protected memory instead of outputting errors. This happens in some of my bigger shaders, maybe the error/warning list is too big and overflows something. I have not seen it fail to compile a valid shader yet.
• Shaders with "discard;" give the error "GL_EXT_demote_to_helper_invocation is only supported in Vulkan GLSL."
• As you mentioned, COLOR0 needs to be SV_TARGET0, but also vertex shaders need to output a SV_POSITION0 and the pixel shader needs to receive it. It compiles either way but the results are terribly broken in the former case. DEPTH to SV_DEPTH also.
• Semantics in general seem to be a bit pickier. You can no longer omit some semantics from the VS/PS parameter list. In MojoShader it was possible to pass a matrix like float4x4 instanceTransform : TEXCOORD1, float otherValue : TEXCOORD5 and this would work, but to get this to work in ShaderConductor I needed to explicitly pass each float4 row individually as TEXCOORD1, 2, 3 and 4.

Thanks for your feedback @minimalism, very much appreciated.

Migrating takes a bit of effort though.

Yeah, a bit of a monkey job. Some automation would be nice.

I noticed allocating a shader off the mainthread (e.g. calling contentManager.ReadAsset<Effect>(pathToShader, null)) gives GL error 1282 (InvalidOperation). IMHO this might be a good change but previously it was wrapped with a Threading.BlockOnUIThread(() => call.

Can you tell me which BlockOnUIThread is not called anymore, that used to be called. I don't think I removed/circumvented any of those. Which GL call is failing exactly?

ShaderConductor.Compile sometimes attempts to access protected memory instead of outputting errors. This happens in some of my bigger shaders, maybe the error/warning list is too big and overflows something. I have not seen it fail to compile a valid shader yet.

That's almost certainly a bug in DirectXShaderCompiler or SPIRV-Cross. If you can send me a crashing shader, I should be able to find out where the crash is happening, and open an issue in the respective repository. Both projects are quite active. I reported a crashing bug before and it was fixed a few weeks later.

Shaders with "discard;" give the error "GL_EXT_demote_to_helper_invocation is only supported in Vulkan GLSL."

I'll see if I can find out more about that in my next ShaderConductor debugging session.

As you mentioned, COLOR0 needs to be SV_TARGET0, but also vertex shaders need to output a SV_POSITION0 and the pixel shader needs to receive it. It compiles either way but the results are terribly broken in the former case. DEPTH to SV_DEPTH also.

Thanks, will add those to the migration document.

Semantics in general seem to be a bit pickier. You can no longer omit some semantics from the VS/PS parameter list. In MojoShader it was possible to pass a matrix like float4x4 instanceTransform : TEXCOORD1, float otherValue : TEXCOORD5 and this would work, but to get this to work in ShaderConductor I needed to explicitly pass each float4 row individually as TEXCOORD1, 2, 3 and 4.

Alright, sounds like something for DirectXShaderCompiler. If the HLSL->DXIL compilation can handle omitting those semantics (not sure if it does) than the HLSL->SPIRV should also handle it.

Can you tell me which BlockOnUIThread is not called anymore, that used to be called. I don't think I removed/circumvented any of those. Which GL call is failing exactly?

The GL calls in PlatformInitialize (and PlatformClear) in ConstantBuffer.OpenGL.cs. On my device, when calling these methods from a background thread GL.GenBuffers will always return -1 and CheckGLError will always throw 1282.
On closer inspection there wasn't a BlockOnUIThread call previously because there weren't any OpenGL calls in these methods.

That's almost certainly a bug in DirectXShaderCompiler or SPIRV-Cross. If you can send me a crashing shader, I should be able to find out where the crash is happening, and open an issue in the respective repository. Both projects are quite active. I reported a crashing bug before and it was fixed a few weeks later.

Sure, this digitalrune shader reproduces the error for me: https://gist.github.com/minimalism/d27e1b20274f0aa646235a9d061401ea
Building this shader on my device causes a System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt at ShaderConductor.Compile.
That shader has an error at ln 184. Replacing that line with the one below it causes the compilation to succeed.

PS Note that this shader also exhibits one of the issues I mentioned above, where the VS outputs { TEXCOORD, SV_POSITION } but the PS parameter list only accepts { TEXCOORD }. This worked in Mojoshader. It compiles fine in ShaderConductor but doesn't seem to render anything to screen.

Hope this is useful :)

The GL calls in PlatformInitialize (and PlatformClear) in ConstantBuffer.OpenGL.cs. On my device, when calling these methods from a background thread GL.GenBuffers will always return -1 and CheckGLError will always throw 1282.

That makes sense, for every ConstantBuffer a uniform buffer object is created. MojoShader didn't need those. I'll just wrap a BlockOnUIThread around these calls then.

I updated #7352
Creating shaders on a background thread sould work now (didn't test it).
Discarding of pixels was fixed with an update of DirectXShaderCompiler.
I also simplified your crashing shader and opened an issue for it.

PS Note that this shader also exhibits one of the issues I mentioned above, where the VS outputs { TEXCOORD, SV_POSITION } but the PS parameter list only accepts { TEXCOORD }

You only mentioned that matrix parameters have to be passed row by row. Do you mean that MojoShader accepted SV_POSITION as a pixel shader parameter, but ShaderConductor doesn't?

Nice! I see they've already merged a fix for the error message error, that will save some frustration for future users no doubt. I'll test the shader on background thread thing even though it's questionable if this practice should really be supported...

You only mentioned that matrix parameters have to be passed row by row. Do you mean that MojoShader accepted SV_POSITION as a pixel shader parameter, but ShaderConductor doesn't?

This situation is what I tried to describe when I said that the pixel shaders needs to receive the SV_POSITION semantic, whereas in MojoShader it was sufficient to just return it from the VS.
Building the same shader with ShaderConductor then inspecting it in NSight revealed that the shader never succeeds in shading any pixels, and always tries to draw off-screen somewhere. When built with MojoShader it would shade the pixels at the position returned by the VS. I don't know which behavior is correct. The shader was originally written by the DigitalRune guys who may have relied on some HLSL/DX9 specific behavior.
To get the shader to behave the same way in ShaderConductor as it did in MojoShader I had to add the SV_POSITION parameter to the pixel shader parameterlist,
I can only speculate as to the reasons, maybe ShaderConductor does some treeshaking and reaches the conclusion that since the pixel shader doesn't receive the SV_POSITION semantic, it can be pruned from the VS.

Nice! I see they've already merged a fix for the error message error, that will save some frustration for future users no doubt.

The PR is now updated and contains this fix.

I'll test the shader on background thread thing even though it's questionable if this practice should really be supported...

I'm pretty sure content loading from multiple threads was already supported in XNA, so MonoGame should do it too.

the pixel shaders needs to receive the SV_POSITION semantic, whereas in MojoShader it was sufficient to just return it from the VS.

I just tested this with a very simple vertex and pixel shader. I don't have that problem. It makes no difference if SV_POSITION is part of the pixel shader parameters or not. Maybe it only shows in more complex shaders, or it has been fixed in the new version.

In MojoShader it was possible to pass a matrix like float4x4 instanceTransform : TEXCOORD1, float otherValue : TEXCOORD5 and this would work, but to get this to work in ShaderConductor I needed to explicitly pass each float4 row individually as TEXCOORD1, 2, 3 and 4.

This too I can't reproduce. It works fine for me. Could the update have fixed this too?

This Pr looks amazing don't give it up!

Closing this PR, as it's contained in #7533