Our Renovate bot pushed the update to NUnit 4.2 earlier this morning, and caused a bunch of tests to fail that worked fine with 4.1.
They either fail with
System.IO.FileNotFoundException : Could not load file or assembly 'System.Buffers, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
at NUnit.Framework.Constraints.Comparers.StreamsComparer.Equal(Object x, Object y, Tolerance& tolerance, ComparisonState state, NUnitEqualityComparer equalityComparer)
at NUnit.Framework.Constraints.NUnitEqualityComparer.AreEqual(Object x, Object y, Tolerance& tolerance, ComparisonState state)
at NUnit.Framework.Constraints.NUnitEqualityComparer.AreEqual(Object x, Object y, Tolerance& tolerance)
at NUnit.Framework.Constraints.EqualConstraint.ApplyTo[TActual](TActual actual)
at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, NUnitString message, String actualExpression, String constraintExpression)
at Project.Tests.Namespace.SomeTests.TestMethod(String data, Boolean expected) in E:\BUILD01A\_work\121\s\Project\Project.Tests\SomeTests.cs:line 129
or
System.IO.FileNotFoundException
HResult=0x80070002
Message=Could not load file or assembly 'System.Memory, Version=4.0.1.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
Source=nunit.framework
StackTrace:
at NUnit.Framework.Constraints.Comparers.StreamsComparer.Equal(Object x, Object y, Tolerance& tolerance, ComparisonState state, NUnitEqualityComparer equalityComparer)
at NUnit.Framework.Constraints.NUnitEqualityComparer.AreEqual(Object x, Object y, Tolerance& tolerance, ComparisonState state)
at NUnit.Framework.Constraints.NUnitEqualityComparer.AreEqual(Object x, Object y, Tolerance& tolerance)
at NUnit.Framework.Constraints.EqualConstraint.ApplyTo[TActual](TActual actual)
at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, NUnitString message, String actualExpression, String constraintExpression)
at Project.Tests.Namespace.SomeTests.TestMethod(String data, Boolean expected) in C:\_git\Project\Project.Tests\SomeTests.cs:line 128
Difference is running on our CI (which has Visual Studio Build Tools 2022 17.6 and uses vstest.console.exe through Azure DevOps Server) vs. running on a local dev machine (which has Visual Studio 2022 17.11 and uses either CodeRush Test Runner or the built-in Test Explorer.) There doesn't appear to be any update to NUnit3TestAdapter, with 4.6.0 being the most recent one.
Same results when running dotnet test in the project directory.
That particular test project is .NET Framework 4.8 with an SDK-style project format (so barely 20 lines of project file with the minimum necessary not provided by various .targets files) while the assembly being tested is .NET Standard 2.0.
The common thing between all failing tests is that they call Is.EqualTo with a parameter or use a Has.Count.EqualTo constraint against a collection:
[Test]
public void TestCollectionCount()
{
var utility = new UtilityClass("something\nto\nparse");
Assert.That(utility.Lines, Has.Count.EqualTo(3)); // throws
//Assert.That(utility.Lines.Count(), Is.EqualTo(3)); // also throws
//ClassicAssert.AreEqual(3, utility.Lines.Count()); // also throws
}
[TestCase(null, false)]
[TestCase("anything", true)]
public void TestCollectionEmpty(string input, bool expected)
{
var utility = new UtilityClass(input);
Assert.That(utility.HasLines, Is.EqualTo(expected)); // throws
}
// with a simplified (and somewhat pointless) test class:
class UtilityClass
{
public UtilityClass(string? lines)
{
Lines = lines?.Split('\n').ToList() ?? [];
HasLines = Lines.Count > 0;
}
public List<string> Lines { get; }
public bool HasLines { get; }
}All of those asserts were converted with NUnit.Analyzers before upgrading to 4.x, they used to be Assert.IsTrue/Assert.IsFalse or Assert.AreEqual.
Replacing the EqualConstraint with something else (for example expected ? Is.True : Is.False rather than using Is.EqualTo(expected)) lets the boolean tests pass; but no alternative seems to work for the count tests.
What's even more surprising: Other Is.EqualTo() (using strings for example) work just fine.
Our Renovate bot pushed the update to NUnit 4.2 earlier this morning, and caused a bunch of tests to fail that worked fine with 4.1.
They either fail with
or
Difference is running on our CI (which has Visual Studio Build Tools 2022 17.6 and uses
vstest.console.exethrough Azure DevOps Server) vs. running on a local dev machine (which has Visual Studio 2022 17.11 and uses either CodeRush Test Runner or the built-in Test Explorer.) There doesn't appear to be any update toNUnit3TestAdapter, with 4.6.0 being the most recent one.Same results when running
dotnet testin the project directory.That particular test project is .NET Framework 4.8 with an SDK-style project format (so barely 20 lines of project file with the minimum necessary not provided by various
.targetsfiles) while the assembly being tested is .NET Standard 2.0.The common thing between all failing tests is that they call
Is.EqualTowith a parameter or use aHas.Count.EqualToconstraint against a collection:All of those asserts were converted with NUnit.Analyzers before upgrading to 4.x, they used to be
Assert.IsTrue/Assert.IsFalseorAssert.AreEqual.Replacing the
EqualConstraintwith something else (for exampleexpected ? Is.True : Is.Falserather than usingIs.EqualTo(expected)) lets the boolean tests pass; but no alternative seems to work for the count tests.What's even more surprising: Other
Is.EqualTo()(using strings for example) work just fine.