Whole tests sequence is aborted when single test fails because of async bug #3930
Description
I had a bugged test on framework 4.8. Simplified reproduction is below.
Obviously, the assertion exception in B will be thrown outside of try-catch in ShouldPass and outside of B test scope.
On fw 4.8 this bug caused to fail only test B. Tests A, C succeeded.
On net 5 this bug causes all tests after B to acquire an inconclusive state. Even all other tests outside of Bar test fixture.
From my perspective, tests isolation has obviously degraded even though it might be due to changes in runtime implementation. Can this be fixed on the level of nunit library in order to roll back to the previous, preferable for user behavior?
[TestFixture]
internal class Bar
{
[TearDown]
public void TearDown()
{
// imitate long cleaning teardown
Thread.Sleep(5.Seconds());
}
[Test]
public void A()
{
Console.WriteLine("Bar.A");
}
[Test]
public void B()
{
ShouldPass(
async () =>
{
// imitate real async work to force context switch
await Task.Yield();
Thread.Sleep(3.Seconds());
Assert.Fail();
});
}
[Test]
public void C()
{
Console.WriteLine("Bar.C");
}
private static void ShouldPass(Action action)
{
var budget = TimeBudget.StartNew(10.Seconds());
while(!budget.HasExpired())
{
try
{
action();
return;
}
catch(AssertionException)
{
}
}
}
}
[TestFixture]
internal class Foo
{
[Test]
public void A()
{
Console.WriteLine("Foo.A");
}
}
tests sequence on fw 4.8
D:\tools\misc\bin\Release>dotnet test misc.exe -v normal
Microsoft (R) Test Execution Command Line Tool Version 16.10.0
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
NUnit Adapter 4.0.0.0: Test execution started
Running all tests in D:\tools\misc\bin\Release\misc.exe
NUnit3TestExecutor discovered 4 of 4 NUnit test cases using Current Discovery mode, Non-Explicit run
Bar.A
Passed A [5 s]
Failed B [5 s]
Stack Trace:
at misc.Bar.<>c.<<B>b__2_0>d.MoveNext() in D:\tools\misc\Program.cs:line 57
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(Object stateMachine)
Bar.C
Passed C [5 s]
Foo.A
NUnit Adapter 4.0.0.0: Test execution complete
Passed A [< 1 ms]
Test Run Failed.
Total tests: 4
Passed: 3
Failed: 1
Total time: 16.3083 Seconds
tests sequence on net 5
D:\tools\misc-core\bin\Release\net5.0>dotnet test misc-core.dll -v normal
Microsoft (R) Test Execution Command Line Tool Version 16.10.0
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
NUnit Adapter 4.0.0.0: Test execution started
Running all tests in D:\tools\misc-core\bin\Release\net5.0\misc-core.dll
NUnit3TestExecutor discovered 4 of 4 NUnit test cases using Current Discovery mode, Non-Explicit run
Bar.A
Passed A [5 s]
Failed B [5 s]
Stack Trace:
at misc_core.Bar.<>c.<<B>b__2_0>d.MoveNext() in D:\tools\misc-core\Program.cs:line 70
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
The active test run was aborted. Reason: Test host process crashed : Unhandled exception. NUnit.Framework.AssertionException
at NUnit.Framework.Assert.ReportFailure(String message)
at NUnit.Framework.Assert.Fail(String message, Object[] args)
at NUnit.Framework.Assert.Fail()
at misc_core.Bar.<>c.<<B>b__2_0>d.MoveNext() in D:\tools\misc-core\Program.cs:line 70
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__140_1(Object state)
at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
at System.Threading.QueueUserWorkItemCallback.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
Test Run Aborted.
Total tests: Unknown
Passed: 1
Failed: 1
Total time: 10.7241 Seconds