Description

The goal of this ticket is to announce the upcoming breaking changes in MSTest v4.

See Migrate from MSTest v3 to MSTest v4

Planned

• Convert MSTest to async pattern (Convert MSTest to async pattern #1284)
  • Youssef note: So far, nothing I see we can improve further.
• TestMethodAttribute.Execute should become TestMethodAttribute.ExecuteAsync #5310
  • User impact: Custom TestMethodAttribute implementations need to change signature to be Async. There is not heavy usage of custom TestMethodAttribute in user codebases, so it's too easy and straightforward to fix (and we can provide automation for it, if needed). This change fixes deadlocks that can happen in some cases due to blocking via GetAwaiter().GetResult()
• Disable appdomain usage by default (Disable appdomain usage by default #1297)
  • Most users will only notice very big performance improvements with this change, without affecting anything for them. In very very niche cases where it affected users, they can add enable it back via runsettings.
• Change ClassCleanup default behavior to be "EndOfClass" #1316 / Remove ClassCleanupBehavior.EndOfAssembly #1574 (PR Remove ClassCleanupBehavior #5827)
  • We already have an analyzer for that with zero user complains in the past. The "default" behavior is almost always incorrect and not what the users want.
• Drop TestTimeout enum and related overload on TimeoutAttribute (Drop TestTimeout enum and related overload on TimeoutAttribute #1317)
  • API that no one uses. Replacement is simple, just replace TestTimeout.Infinite with int.MaxValue.
• Update TestContext.Properties return type to IDictionary<string, object> (Update TestContext.Properties return type to IDictionary<string, object> #1377)
  • Small impact on users.
• [Breaking][v4] Assembly version shouldn't be fixed to 14.0.0.0 #5690 (kinda linked to Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices errors due to path length not handled well on UWP #1607) - also let's change executor uri.
  • In most cases this will be transparent for users, but has high impact on reliability of MSTest.
• [v4] Broader changes for the story around attributes and their inheritance #4292
• [MSTest v4][breaking] Use CallerFilePath and CallerLineNumber in TestMethodAttribute #5500
• Return T from Assert.IsInstanceOfType<T>() #1573
• [Breaking] Remove ITestMethod interface in platform services in v4 #4724
• Reading runsettings shouldn't use ReadToFollowing #5800
• [MSTest v4] Consider removing CssIterationAttribute, CssProjectStructureAttribute, and DescriptionAttribute #5787
• DataRow: Nulls and empty string are displayed as no data in display name (DataRow: Nulls and empty string are displayed as no data in display name #1647) ([Breaking][v4] Remove TestIdGenerationStrategy #5757)
• Remove deprecated TestContext properties that were deprecated as part of Deprecate TestContext properties kept for compatibility #1453 - [Breaking][v4] Remove deprecated TestContext properties #5759
• MSTest v4 TestContext properties should not be nullable #1403
• Add relaxed versions of Assert.ThrowsException and Assert.ThrowsExceptionAsync #4257 (comment)
• Deprecate ExpectedExceptionAttribute (Deprecate ExpectedException attribute. #135) - Bump analyzer in 3.10 to be warning, and drop in v4
• [v4][Breaking] Review analyzer serverities #5748
• Use a better serialization/deserialization mechanism for data discovery (Use a better serialization/deserialization mechanism for data discovery #1462) (Related PR: Adjust TestCase.Id to account for randomized tests (include test case index instead of serialized data) #5871)
• [MSTest][v4] STAThreadAttribute should be sealed #5693
• Make UseCooperativeCancellationForTimeoutAnalyzer warning #5879
• Drop support for < net8.0 in MSTest and MTP #5614
• Move unfolding strategy from data sources to test method #5876
• Include parameter types in test id #5899
• Don't use VSTest's TestIdProvider for generating TestCase.Id (Few adjustments for test id #5936)
• [Breaking][v4] MSTest.TestAdapter (and consequently MSTest metapackage) should drop netstandard2.0 #5645
• Consider renaming ConditionBaseAttribute.ShouldRun to IsConditionMet/IsConditionSatisfied #6086
• [Breaking][v4] Don't ignore ReflectionTypeLoadExceptions #5680
• TreatDiscoveryWarningsAsErrors should throw instead of sending message #5992
• Caller argument expression (Openness for Caller Argument Expression in Assert failure messages #1154)
  • Part of the work is being done in Add CallerArgumentExpression overloads to Assert.IsTrue/IsFalse/IsNull/IsNotNull #5851
  • [rel/4.0] Complete caller argument expression for all assertions #6329

To discuss

• DisableAppDomain for MSTest v4 vstest#15220

Follow-ups

After merging to main to reduce conflicts

• Cleanup resource files (resx). Cleanup resources #6656
• Consolidate MicrosoftTestingTargetFrameworks and SupportedNetFrameworks Cleanup MicrosoftTestingTargetFrameworks #6657

Any-time, early if possible (necessary or helps with migration)

• Analyzers for the caller info changes of TestMethodAttribute
  • Warn when a single string argument is explicitly passed to TestMethodAttribute. This likely used to mean "display name". (Implement analyzer for TestMethod display name #6403)
  • When inheriting the attribute, ensure that there is a constructor that propagates caller info. (Implement analyzer for TestMethodAttribute inheritance #6405)
• Add CodeFixProvider for migration tasks from v3 to v4
  • A codefix provider for [ClassCleanup(ClassCleanupBehavior.EndOfClass)] to remove the ClassCleanupBehavior as the enum was removed, and now EndOfClass is the behavior of ClassCleanup. (Implement fixer for ClassCleanupBehavior #6406)
  • A codefix provider to help migrating Assert.IsInstanceOfType<T> with out parameter (Implement CodeFixProvider for moving over IsInstanceOf breaking change #6410)
    • public static void IsInstanceOfType<T>([NotNull] object? value, out T instance)
    • public static void IsInstanceOfType<T>([NotNull] object? value, out T instance, string? message)
    • public static void IsInstanceOfType<T>([NotNull] object? value, out T instance, [InterpolatedStringHandlerArgument(nameof(value))] ref AssertGenericIsInstanceOfTypeInterpolatedStringHandler<T> message)
  • A codefix provider for overriding Execute of TestMethodAttribute to switch that to ExecuteAsync. (Implement codefix to switch Execute to ExecuteAsync #6414)
  • A codefix provider to account for the TestContext.Properties change from IDictionary to IDictionary<TKey, TValue> (e.g, switch Contains to ContainsKey) (Implement codefix for migrating TestContext.Properties to generic IDictionary #6415)
• New version of Microsoft.Playwright.MSTest

Any-time, lower value

• Warning for no longer supported ClassCleanupLifecycle in runsettings/testconfig.json
• Add analyzer for Console.Write**, Trace.Write**, and Debug.Write** usage in tests to suggest using TestContext instead. #6285
• Warn when the attribute isn't syntactically declared just above the method. Having the attribute declared just above the method helps caller info be as close as possible to the method declaration.

Delayed

• Consider enforcing MapInconclusiveToFailed in MTP #5584
• Timeout should produce dangling activities by default #2433
• Disable CaptureTraceOutput by default (linked issue TraceListenerManager is not thread safe #2616) ([Breaking] Disable CaptureTraceOutput by default #5756)
• [Breaking] Remove [DataTestMethod] #4166 (reverted in [rel/4.0] Revert "Drop DataTestMethodAttribute (#5831)" #6091, postponing to v5)
• Waiting for user feedback if needed to avoid unnecessary maintenance cost of the codefixes
  • A codefix provider to help moving unfolding strategy to TestMethodAttribute (This one is low value IMO)
  • A codefix provider to account for the drop of TestTimeout so that it switches TestTimeout.Infinite to int.MaxValue (This one is low value IMO)
  • A codefix provider to help migrating users affected by the removal of IEquatable<T> Assert.AreEqual API (This one is low value IMO).

Canceled

• Better DisplayName for data tests (Better DisplayNames for DataSource tests #713)
• Reintroduce tree structure (InnerResults) for parameterized tests (Data Driven tests - Individual results of data rows are not in tree structure, missing parentID in trx file #1024)
• [MSTest v4]: Discuss if MSTest.Sdk should set MSTestAnalysisMode to Recommended #4974
• [MSTest v4]: Discuss if the default of TestingPlatformDotnetTestSupport should match EnableMSTestRunner #4973 (we want to break the old dotnet test exp for MTP altogether, so this will become irrelevant)
• Remove Assert.Equals (Obsolete Assert.Equals, and add an obsolete ReferenceEquals #5758)
• Separate GetDisplayName out of ITestDataSource #5307
• Discuss whether TestMethodAttribute.Execute should return a single TestResult object #5309
• Align naming used for attributes/types allowing to control some behavior #1327
• [MSTest v4] Fail when discovery fails on a class #5327 (no longer needed)

Done in v3 already

• Remove deprecated BeginTimer and EndTimer methods (Remove deprecated BeginTimer and EndTimer methods #1454)
• Deprecate TestContext properties kept for compatibility (Deprecate TestContext properties kept for compatibility #1453)
• Unify SetOutcome behavior across all target frameworks (TestContext: Unify SetOutcome behavior across TFMs #1426)