Casts from ulong to int changed behaviour in .NET Core 3.0

We are updating an existing codebase to target multiple frameworks (net472 plus the active LTS version of .NET).
We ran into odd behaviour with code that interpreted bytes as a big-endian two's complement integer.
It turns out conversions from ulong to int behave differently depending on whether it's one big expression, or uses an intermediate variable.

var bytes = new byte[] { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xfe };

var direct = (int) ((((ulong)(bytes[0])) << 56) |
                    (((ulong)(bytes[1])) << 48) |
                    (((ulong)(bytes[2])) << 40) |
                    (((ulong)(bytes[3])) << 32) |
                    (((ulong)(bytes[4])) << 24) |
                    (((ulong)(bytes[5])) << 16) |
                    (((ulong)(bytes[6])) << 8) |
                    (((ulong)(bytes[7])) << 0));
var intermediate = (((ulong)(bytes[0])) << 56) |
                   (((ulong)(bytes[1])) << 48) |
                   (((ulong)(bytes[2])) << 40) |
                   (((ulong)(bytes[3])) << 32) |
                   (((ulong)(bytes[4])) << 24) |
                   (((ulong)(bytes[5])) << 16) |
                   (((ulong)(bytes[6])) << 8) |
                   (((ulong)(bytes[7])) << 0);
var cast = (int) intermediate;

Console.WriteLine($"Bytes: {BitConverter.ToString(bytes)}");
Console.WriteLine($"Direct Conversion: {direct}");
Console.WriteLine($"Conversion Using Intermediate: {intermediate} -> {cast}");

(I'm sure this code could be written much better - but it's the real-world code that showed the issue for us.)
The expected result is -2, regardless of which approach is taken.

Configuration

After detecting this when targeting netcoreapp3.1, I set up the above example as a console app, with

    <TargetFrameworks>net40;net472;netcoreapp2.1;netcoreapp2.2;netcoreapp3.0;netcoreapp3.1;net5.0</TargetFrameworks>
    <CheckEolTargetFramework>false</CheckEolTargetFramework>

This shows:

Debug/Release configuration makes no difference.
intermediate always contains 18446744073709551614.

Regression?

The behaviour in .NET Framework and .NET Core 2.x seems sensible.
The new behaviour is decidedly odd and unexpected, so I would class this as a regression.

Other information

While there is a "workaround" of sorts (using a variable to store the computed ulong, which is also useful for debugging), it's a very subtle issue, and there is no diagnostic or runtime error, making places where such a workaround should be applied non-trivial to find.