Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests are extremely flaky #52802
Description
Different, but big number of failures reported on each run. Failed test pass on rerun.
For example, first run:
[xUnit.net 01:09:46.78] Finished: Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests
========== Test run finished: 24373 Tests (23189 Passed, 1011 Failed, 173 Skipped) run in 70.2 min ==========
Rerun of failed tests:
[xUnit.net 00:07:38.94] Finished: Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests
========== Test run finished: 1011 Tests (1011 Passed, 0 Failed, 0 Skipped) run in 7.7 min ==========
Typical failures look like this:
Microsoft.CodeAnalysis.Editor.CSharp.UnitTests.UseExplicitTypeForConst.UseExplicitTypeForConstTests.TestWithNonConstantTupleType
158ms
Error: Actual and expected values differ. Expected shown in baseline of diff:
class C
{
void M()
{
- const (int, bool) v = (0, true);
+ const (System.Int32, System.Boolean) v = (0, true);
}
}
Expected: True
Actual: False
Another type of failures looks like this. Even though a base line diff is reported, I cannot see what is different between the lines at the beginning of the class declaration. Perhaps the diff is in whitespace/line breaks.
Microsoft.CodeAnalysis.Editor.CSharp.UnitTests.UsePatternCombinators.CSharpUsePatternCombinatorsDiagnosticAnalyzerTests.TestOnExpression(expression: "ch < ' ' || ch >= 0x100 || 'a' == ch", expected: "ch is < ' ' or >= (char)0x100 or 'a'")
1s 28ms
Error: Actual and expected values differ. Expected shown in baseline of diff:
using System;
using System.Collections.Generic;
class C
{
- static bool field = ch is < ' ' or >= (char)0x100 or 'a';
- static bool Method() => ch is < ' ' or >= (char)0x100 or 'a';
- static bool Prop1 => ch is < ' ' or >= (char)0x100 or 'a';
- static bool Prop2 { get; } = ch is < ' ' or >= (char)0x100 or 'a';
- static void If() { if (ch is < ' ' or >= (char)0x100 or 'a') ; }
- static void Argument1() => Test(ch is < ' ' or >= (char)0x100 or 'a');
- static void Argument2() => Test(() => ch is < ' ' or >= (char)0x100 or 'a');
- static void Argument3() => Test(_ => ch is < ' ' or >= (char)0x100 or 'a');
+ static bool field = ch is < ' ' or >= (Char)0x100 or 'a';
+ static bool Method() => ch is < ' ' or >= (Char)0x100 or 'a';
+ static bool Prop1 => ch is < ' ' or >= (Char)0x100 or 'a';
+ static bool Prop2 { get; } = ch is < ' ' or >= (Char)0x100 or 'a';
+ static void If() { if (ch is < ' ' or >= (Char)0x100 or 'a') ; }
+ static void Argument1() => Test(ch is < ' ' or >= (Char)0x100 or 'a');
+ static void Argument2() => Test(() => ch is < ' ' or >= (Char)0x100 or 'a');
+ static void Argument3() => Test(_ => ch is < ' ' or >= (Char)0x100 or 'a');
static void Test(bool b) {}
static void Test(Func<bool> b) {}
static void Test(Func<object, bool> b) {}
- static void For() { for (; ch is < ' ' or >= (char)0x100 or 'a'; ); }
- static void Local() { var local = ch is < ' ' or >= (char)0x100 or 'a'; }
- static void Conditional() { _ = ch is < ' ' or >= (char)0x100 or 'a' ? ch is < ' ' or >= (char)0x100 or 'a' : ch is < ' ' or >= (char)0x100 or 'a'; }
- static void Assignment() { _ = ch is < ' ' or >= (char)0x100 or 'a'; }
- static void Do() { do ; while (ch is < ' ' or >= (char)0x100 or 'a'); }
- static void While() { while (ch is < ' ' or >= (char)0x100 or 'a') ; }
- static bool When() => o switch { _ when ch is < ' ' or >= (char)0x100 or 'a' => ch is < ' ' or >= (char)0x100 or 'a' };
- static bool Return() { return ch is < ' ' or >= (char)0x100 or 'a'; }
- static IEnumerable<bool> YieldReturn() { yield return ch is < ' ' or >= (char)0x100 or 'a'; }
- static Func<object, bool> SimpleLambda() => o => ch is < ' ' or >= (char)0x100 or 'a';
- static Func<bool> ParenthesizedLambda() => () => ch is < ' ' or >= (char)0x100 or 'a';
- static void LocalFunc() { bool LocalFunction() => ch is < ' ' or >= (char)0x100 or 'a'; }
+ static void For() { for (; ch is < ' ' or >= (Char)0x100 or 'a'; ); }
+ static void Local() { var local = ch is < ' ' or >= (Char)0x100 or 'a'; }
+ static void Conditional() { _ = ch is < ' ' or >= (Char)0x100 or 'a' ? ch is < ' ' or >= (Char)0x100 or 'a' : ch is < ' ' or >= (Char)0x100 or 'a'; }
+ static void Assignment() { _ = ch is < ' ' or >= (Char)0x100 or 'a'; }
+ static void Do() { do ; while (ch is < ' ' or >= (Char)0x100 or 'a'); }
+ static void While() { while (ch is < ' ' or >= (Char)0x100 or 'a') ; }
+ static bool When() => o switch { _ when ch is < ' ' or >= (Char)0x100 or 'a' => ch is < ' ' or >= (Char)0x100 or 'a' };
+ static bool Return() { return ch is < ' ' or >= (Char)0x100 or 'a'; }
+ static IEnumerable<bool> YieldReturn() { yield return ch is < ' ' or >= (Char)0x100 or 'a'; }
+ static Func<object, bool> SimpleLambda() => o => ch is < ' ' or >= (Char)0x100 or 'a';
+ static Func<bool> ParenthesizedLambda() => () => ch is < ' ' or >= (Char)0x100 or 'a';
+ static void LocalFunc() { bool LocalFunction() => ch is < ' ' or >= (Char)0x100 or 'a'; }
static int i;
static int? nullable;
static object o;
static char ch;
}
Expected: True
Actual: False
I am not sure what is going on here. One theory is, perhaps, the tests are some how depended, i.e. relying on some shared data (whatever that might be) and there is a "race" to initialize/modify the data. There might be some dependency on environment, etc.